Systems and methods for sign language recognition

ABSTRACT

A sensory eyewear system for a mixed reality device can facilitate user&#39;s interactions with the other people or with the environment. As one example, the sensory eyewear system can recognize and interpret a sign language, and present the translated information to a user of the mixed reality device. The wearable system can also recognize text in the user&#39;s environment, modify the text (e.g., by changing the content or display characteristics of the text), and render the modified text to occlude the original text.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/801,684, filed on Feb. 26, 2020, entitled “SYSTEMS AND METHODS FORSIGN LANGUAGE RECOGNITION,” which is a continuation of U.S. patentapplication Ser. No. 15/702,312, filed on Sep. 12, 2017, entitled“SYSTEMS AND METHODS FOR SIGN LANGUAGE RECOGNITION,” which claims thebenefit of priority under 35 U.S.C. § 119(e) to U.S. ProvisionalApplication No. 62/394,013, filed on Sep. 13, 2016, entitled “SENSORYEYEWEAR,” and U.S. Provisional Application No. 62/440,320 filed on Dec.29, 2016, entitled “SYSTEMS AND METHODS FOR AUGMENTED REALITY,” thedisclosures of each of which are hereby incorporated by reference hereinin their entireties.

FIELD

The present disclosure relates to virtual reality and augmented realityimaging and visualization systems and more particularly to recognizingsign language or text in an environment and rendering virtual contentbased on the recognized sign language or text.

BACKGROUND

Modern computing and display technologies have facilitated thedevelopment of systems for so called “virtual reality”, “augmentedreality”, or “mixed reality” experiences, wherein digitally reproducedimages or portions thereof are presented to a user in a manner whereinthey seem to be, or may be perceived as, real. A virtual reality, or“VR”, scenario typically involves presentation of digital or virtualimage information without transparency to other actual real-world visualinput; an augmented reality, or “AR”, scenario typically involvespresentation of digital or virtual image information as an augmentationto visualization of the actual world around the user; a mixed reality,or “MR”, related to merging real and virtual worlds to produce newenvironments where physical and virtual objects co-exist and interact inreal time. As it turns out, the human visual perception system is verycomplex, and producing a VR, AR, or MR technology that facilitates acomfortable, natural-feeling, rich presentation of virtual imageelements amongst other virtual or real-world imagery elements ischallenging. Systems and methods disclosed herein address variouschallenges related to VR, AR and MR technology.

SUMMARY

Various embodiments of a mixed reality system for recognizing signlanguage and text in an environment are disclosed. These embodimentsadvantageously may permit greater interaction among differently-abledpersons.

A sensory eyewear system for a mixed reality device can facilitateuser's interactions with the other people or with the environment. Asone example, the sensory eyewear system can recognize and interpret asign language, and present the translated information to a user of themixed reality device. The wearable system can also recognize text in theuser's environment, modify the text (e.g., by changing the content ordisplay characteristics of the text), and render the modified text toocclude the original text.

Details of one or more implementations of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages will becomeapparent from the description, the drawings, and the claims. Neitherthis summary nor the following detailed description purports to defineor limit the scope of the inventive subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an illustration of a mixed reality scenario with certainvirtual reality objects, and certain physical objects viewed by aperson.

FIG. 2A schematically illustrates an example of a wearable system whichcan implement a sensory eyewear system.

FIG. 2B schematically illustrates various example components of awearable system.

FIG. 3 schematically illustrates aspects of an approach for simulatingthree-dimensional imagery using multiple depth planes.

FIG. 4 schematically illustrates an example of a waveguide stack foroutputting image information to a user.

FIG. 5 shows example exit beams that may be outputted by a waveguide.

FIG. 6 is a schematic diagram showing an optical system including awaveguide apparatus, an optical coupler subsystem to optically couplelight to or from the waveguide apparatus, and a control subsystem, usedin the generation of a multi-focal volumetric display, image, or lightfield.

FIG. 7 is a block diagram of an example of a wearable system.

FIG. 8 is a process flow diagram of an example of a method of renderingvirtual content in relation to recognized objects.

FIG. 9 is a block diagram of another example of a wearable system thatincludes a sensory eyewear system.

FIG. 10 is a process flow diagram of an example of a method fordetermining user input to a wearable system.

FIG. 11 is a process flow diagram of an example of a method forinteracting with a virtual user interface.

FIG. 12 schematically illustrates an overall system view depictingmultiple wearable systems interacting with each other.

FIG. 13A shows an example user experience of a sensory eyewear systemwhere the sensory eyewear system can interpret a sign language (e.g.,gestured by a signer).

FIG. 13B shows another example user experience of a sensory eyewearsystem, where target speech and auxiliary information are bothpresented.

FIG. 13C shows an example user experience of a sensory eyewear system ina telepresence session.

FIG. 13D illustrates an example virtual user interface for interpretinga sign language.

FIGS. 14A and 14B illustrate example processes for facilitatinginterpersonal communications with a sensory eyewear system.

FIG. 14C is a process flow diagram of an example method for determiningauxiliary information and presenting the auxiliary informationassociated with converted text.

FIG. 15 illustrates another example process for facilitatinginterpersonal communications with a sensory eyewear system.

FIGS. 16A-16E illustrate example user experiences for a sensory eyewearsystem which is configured to recognize text in the environment, modifythe display characteristics of the text, and render the modified text.

FIG. 17 illustrates an example process of a sensory eyewear forfacilitating a user's interactions with the environment.

FIG. 18 illustrates an example of assisting a user in understandingsignage in a physical environment by modifying the content of thesignage.

FIG. 19 illustrates an example process of assisting a user inunderstanding signage in a physical environment.

Throughout the drawings, reference numbers may be re-used to indicatecorrespondence between referenced elements. The drawings are provided toillustrate example embodiments described herein and are not intended tolimit the scope of the disclosure.

DETAILED DESCRIPTION Overview

A wearable system which is configured to present AR/VR/MR content canimplement an sensory eyewear system to enhance the user's interactionwith other people or the environment. An example wearable system cancomprise a head-mounted display, various imaging sensors, and one ormore hardware processors. The display can be a see-through display wornin front of the eye or eyes.

To enhance the user's interaction experience with other people, thewearable system can be configured to capture and interpret a signlanguage. A sign language primarily uses visual gestures (e.g., handshapes; hand orientations; hand, arm, or body movements; or facialexpressions) to communicate. There are hundreds of sign languages usedaround the world. Some sign languages may be used more often thanothers. For example, American sign language (ASL) is widely used in theU.S. and Canada.

Many people do not know any sign languages. A speech- orhearing-challenged person and a conversation partner may not be familiarwith the same sign language. This can impede conversation with thehearing-challenged or the speech-challenged persons. Accordingly, thewearable system that can image signs (e.g., gestures) being made by aconversation partner, convert the signs to text or graphics (e.g.,graphics of sign language gestures in a sign language the system userunderstands), and then display information associated with the sign(e.g., a translation of the signs into a language understood by theuser) would greatly help improve the communication between the user andthe conversation partner. Further, it may be desirable to have awearable system which can provide textual or graphical conversion of asign language in (or near) real-time with a minimal level of distractionto, and an insignificant level of effort by, a user of the wearablesystem.

The present disclosure discloses examples of such desirable systems inthe context of a wearable device. The wearable device may include ahead-mounted component (such as, e.g., a head-mounted display). Such adevice can allow a user to visually receive information which isprovided by a computing device in such a manner that the information issimultaneously viewable alongside (or on top of) the normally viewablereal world. Such a system can be used to display any form of informationthat can be displayed on a traditional computer screen such ascharacters, image effects, text, graphics, or video of any kind.

The wearable system described herein can combine sign languagerecognition (SLR) and display capability of a wearable device to providea user with information based on a detected sign language. For example,an outward-facing camera on the wearable device can image gestures beingmade, identify signs among the gestures, translate the signs to alanguage the user understands, and display the translation to the user.A transcript (e.g., a caption or a text bubble) of the detected signlanguage can be displayed to the user by the wearable system. A machinelearning algorithm (e.g., a deep neural network) can receive the imagesand perform the identification and translation of the signs. Whenprompted by the user, the meaning of a word in the transcript orrelevant information from an appropriate source can be displayed. Thekinds of auxiliary information that the wearable system can provide canbe as unlimited as the vast array of available information resources,e.g., on the Internet.

In addition to or in alternative to enhancing the user's interactionexperience with other people, the sensory eyewear system can alsoimprove the user's experience with the environment. As an example ofimproving user interactions with the environment, a wearable systemimplementing the sensory eyewear system can recognize text (e.g., texton signage such as, e.g., commercial or public display signs) in anenvironment, modify the display characteristics of the text (e.g., byincreasing the size of the text) or modify the content of the text(e.g., by translating the text to another language), and render themodified text over the physical text in the environment.

As further described herein, a wearable system can receive an image ofthe user's environment. The image may be acquired by the outward-facingimaging system of a wearable device or a totem associated with thewearable device. The wearable system can determine whether the imagecomprises one or more letters or characters and convert the one or moreletters or characters into text. The wearable system may determinewhether the image comprises letters or characters using a variety oftechniques, such as, for example, machine learning algorithms or opticalcharacter recognition (OCR) algorithms. The wearable system may useobject recognizers (e.g., described in FIG. 7) to identify the lettersand characters and convert them into text.

In certain embodiments, the text can be displayed for the userdifferently than the user would see without the wearable device. Forexample, the wearable system can cause a head-mounted dispaly to displaythe text in a font size that is different from a font size associatedwith the letters or characters associated with the original image. Thewearable system can also improve the display quality of the text. Forexample, various environmental factors, such as fog, haze, rain, brightlight, low light, low light or color contrast between the letters andthe surrounding image, etc., can impede a user's clear view of text inthe environment without the wearable system. The wearable system maypresent a sign (e.g., with increased contrast ratio or larger font) thatwill increase the clarity of the text.

The wearable system can also translate the text (e.g., the text onsignage) from its original language to a target language. For example,the text may be translated from a language that the user does notunderstand to a language that the user understands. The translated textmay be rendered over the original text such that the user can readilyview the text in a language that the user is able to understand.

Examples of 3D Display of a Wearable System

A wearable system (also referred to herein as an augmented reality (AR)system) can be configured to present 2D or 3D virtual images to a user.The images may be still images, frames of a video, or a video, incombination or the like. At least a portion of the wearable system canbe implemented on a wearable device that can present a VR, AR, or MRenvironment, alone or in combination, for user interaction. The wearabledevice can be a head-mounted device (HMD) which is used interchangeablyas an AR device (ARD). Further, for the purpose of the presentdisclosure, the term “AR” is used interchangeably with the term “MR”.

FIG. 1 depicts an illustration of a mixed reality scenario with certainvirtual reality objects, and certain physical objects viewed by aperson. In FIG. 1, an MR scene 100 is depicted wherein a user of an MRtechnology sees a real-world park-like setting 110 featuring people,trees, buildings in the background, and a concrete platform 120. Inaddition to these items, the user of the MR technology also perceivesthat he “sees” a robot statue 130 standing upon the real-world platform120, and a cartoon-like avatar character 140 flying by which seems to bea personification of a bumble bee, even though these elements do notexist in the real world.

In order for the 3D display to produce a true sensation of depth, andmore specifically, a simulated sensation of surface depth, it may bedesirable for each point in the display's visual field to generate anaccommodative response corresponding to its virtual depth. If theaccommodative response to a display point does not correspond to thevirtual depth of that point, as determined by the binocular depth cuesof convergence and stereopsis, the human eye may experience anaccommodation conflict, resulting in unstable imaging, harmful eyestrain, headaches, and, in the absence of accommodation information,almost a complete lack of surface depth.

VR, AR, and MR experiences can be provided by display systems havingdisplays in which images corresponding to a plurality of depth planesare provided to a viewer. The images may be different for each depthplane (e.g., provide slightly different presentations of a scene orobject) and may be separately focused by the viewer's eyes, therebyhelping to provide the user with depth cues based on the accommodationof the eye required to bring into focus different image features for thescene located on different depth plane or based on observing differentimage features on different depth planes being out of focus. Asdiscussed elsewhere herein, such depth cues provide credible perceptionsof depth.

FIG. 2A illustrates an example of wearable system 200 which can beconfigured to provide an AR/VR/MR scene. The wearable system 200 canalso be referred to as the AR system 200. The wearable system 200includes a display 220, and various mechanical and electronic modulesand systems to support the functioning of display 220. The display 220may be coupled to a frame 230, which is wearable by a user, wearer, orviewer 210. The display 220 can be positioned in front of the eyes ofthe user 210. The display 220 can present AR/VR/MR content to a user.The display 220 can comprise a head mounted display that is worn on thehead of the user. In some embodiments, a speaker 240 is coupled to theframe 230 and positioned adjacent the ear canal of the user (in someembodiments, another speaker, not shown, is positioned adjacent theother ear canal of the user to provide for stereo/shapeable soundcontrol). The display 220 can include an audio sensor (e.g., amicrophone) 232 for detecting an audio stream from the environment andcapture ambient sound. In some embodiments, one or more other audiosensors, not shown, are positioned to provide stereo sound reception.Stereo sound reception can be used to determine the location of a soundsource. The wearable system 200 can perform voice or speech recognitionon the audio stream.

The wearable system 200 can include an outward-facing imaging system 464(shown in FIG. 4) which observes the world in the environment around theuser. The wearable system 200 can also include an inward-facing imagingsystem 462 (shown in FIG. 4) which can track the eye movements of theuser. The inward-facing imaging system may track either one eye'smovements or both eyes' movements. The inward-facing imaging system 462may be attached to the frame 230 and may be in electrical communicationwith the processing modules 260 or 270, which may process imageinformation acquired by the inward-facing imaging system to determine,e.g., the pupil diameters or orientations of the eyes, eye movements oreye pose of the user 210.

As an example, the wearable system 200 can use the outward-facingimaging system 464 or the inward-facing imaging system 462 to acquireimages of a pose of the user. The images may be still images, frames ofa video, or a video.

The display 220 can be operatively coupled 250, such as by a wired leador wireless connectivity, to a local data processing module 260 whichmay be mounted in a variety of configurations, such as fixedly attachedto the frame 230, fixedly attached to a helmet or hat worn by the user,embedded in headphones, or otherwise removably attached to the user 210(e.g., in a backpack-style configuration, in a belt-coupling styleconfiguration).

The local processing and data module 260 may comprise a hardwareprocessor, as well as digital memory, such as non-volatile memory (e.g.,flash memory), both of which may be utilized to assist in theprocessing, caching, and storage of data. The data may include data a)captured from sensors (which may be, e.g., operatively coupled to theframe 230 or otherwise attached to the user 210), such as image capturedevices (e.g., cameras in the inward-facing imaging system or theoutward-facing imaging system), audio sensors (e.g., microphones),inertial measurement units (IMUs), accelerometers, compasses, globalpositioning system (GPS) units, radio devices, or gyroscopes; or b)acquired or processed using remote processing module 270 or remote datarepository 280, possibly for passage to the display 220 after suchprocessing or retrieval. The local processing and data module 260 may beoperatively coupled by communication links 262 or 264, such as via wiredor wireless communication links, to the remote processing module 270 orremote data repository 280 such that these remote modules are availableas resources to the local processing and data module 260. In addition,remote processing module 280 and remote data repository 280 may beoperatively coupled to each other.

In some embodiments, the remote processing module 270 may comprise oneor more processors configured to analyze and process data or imageinformation. In some embodiments, the remote data repository 280 maycomprise a digital data storage facility, which may be available throughthe internet or other networking configuration in a “cloud” resourceconfiguration. In some embodiments, all data is stored and allcomputations are performed in the local processing and data module,allowing fully autonomous use from a remote module.

FIG. 2B shows the wearable system 200 which can include a display 220and a frame 230. A blown-up view 202 schematically illustrates variouscomponents of the wearable system 200. In certain implements, one ormore of the components illustrated in FIG. 2B can be part of the display220. The various components alone or in combination can collect avariety of data (such as e.g., audio or visual data) associated with theuser of the wearable system 200 or the user's environment. It should beappreciated that other embodiments may have additional or fewercomponents depending on the application for which the wearable system isused. Nevertheless, FIG. 2B provides a basic idea of some of the variouscomponents and types of data that may be collected, analyzed, and storedthrough the wearable system.

FIG. 2B shows an example wearable system 200 which can include thedisplay 220. The display 220 can comprise a display lens 106 that may bemounted to a user's head or a housing or frame 108, which corresponds tothe frame 230. The display lens 106 may comprise one or more transparentmirrors positioned by the housing 108 in front of the user's eyes 302,304 and may be configured to bounce projected light 38 into the eyes302, 304 and facilitate beam shaping, while also allowing fortransmission of at least some light from the local environment. Thewavefront of the projected light beam 38 may be bent or focused tocoincide with a desired focal distance of the projected light. Asillustrated, two wide-field-of-view machine vision cameras 16 (alsoreferred to as world cameras) can be coupled to the housing 108 to imagethe environment around the user. These cameras 16 can be dual capturevisible light/non-visible (e.g., infrared) light cameras. The cameras 16may be part of the outward-facing imaging system 464 shown in FIG. 4.Image acquired by the world cameras 16 can be processed by the poseprocessor 36. For example, the pose processor 36 can implement one ormore object recognizers 708 (e.g., shown in FIG. 7) to identify a poseof a user or another person in the user's environment or to identify aphysical object in the user's environment.

With continued reference to FIG. 2B, a pair of scanned-lasershaped-wavefront (e.g., for depth) light projector modules with displaymirrors and optics configured to project light 38 into the eyes 302, 304are shown. The depicted view also shows two miniature infrared cameras24 paired with infrared light sources 26 (such as light emitting diodes“LED”s), which are configured to be able to track the eyes 302, 304 ofthe user to support rendering and user input. The cameras 24 may be partof the inward-facing imaging system 462 shown in FIG. 4 The wearablesystem 200 can further feature a sensor assembly 39, which may compriseX, Y, and Z axis accelerometer capability as well as a magnetic compassand X, Y, and Z axis gyro capability, preferably providing data at arelatively high frequency, such as 200 Hz. The sensor assembly 39 may bepart of the IMU described with reference to FIG. 2A The depicted system200 can also comprise a head pose processor 36, such as an ASIC(application specific integrated circuit), FPGA (field programmable gatearray), or ARM processor (advanced reduced-instruction-set machine),which may be configured to calculate real or near-real time user headpose from wide field of view image information output from the capturedevices 16. The head pose processor 36 can be a hardware processor andcan be implemented as part of the local processing and data module 260shown in FIG. 2A.

Also shown is a processor 32 configured to execute digital or analogprocessing to derive pose from the gyro, compass, or accelerometer datafrom the sensor assembly 39. The processor 32 may be part of the localprocessing and data module 260 shown in FIG. 2A. The wearable system 200as shown in FIG. 2B can also include a position system such as, e.g., aGPS 37 (global positioning system) to assist with pose and positioninganalyses. In addition, the GPS may further provide remotely-based (e.g.,cloud-based) information about the user's environment. This informationmay be used for recognizing objects or information in user'senvironment.

The wearable system may combine data acquired by the GPS 37 and a remotecomputing system (such as, e.g., the remote processing module 270,another user's ARD, etc.) which can provide more information about theuser's environment. As one example, the wearable system can determinethe user's location based on GPS data and retrieve a world map (e.g., bycommunicating with a remote processing module 270) including virtualobjects associated with the user's location. As another example, thewearable system 200 can monitor the environment using the world cameras16 (which may be part of the outward-facing imaging system 464 shown inFIG. 4). Based on the images acquired by the world cameras 16, thewearable system 200 can detect characters in the environment (e.g., byusing one or more object recognizers 708 shown in FIG. 7). The wearablesystem can further use data acquired by the GPS 37 to interpret thecharacters. For example, the wearable system 200 can identify ageographic region where the characters are located and identify one ormore languages associated with the geographic region. The wearablesystem can accordingly interpret the characters based on the identifiedlanguage(s), e.g., based on syntax, grammar, sentence structure,spelling, punctuation, etc., associated with the identified language(s).In one example, a user 210 in Germany can perceive a traffic sign whiledriving down the autobahn. The wearable system 200 can identify that theuser 210 is in Germany and that the text from the imaged traffic sign islikely in German based on data acquired from the GPS 37 (alone or incombination with images acquired by the world camera 16).

In some situations, the images acquired by the world cameras 16 mayinclude incomplete information of an object in a user's environment. Forexample, the image may include an incomplete text (e.g., a sentence, aletter, or a phrase) due to a hazy atmosphere, a blemish or error in thetext, low lighting, fuzzy images, occlusion, limited FOV of the worldcameras 16, etc. The wearable system 200 could use data acquired by theGPS 37 as a context clue in recognizing the text in image.

The wearable system 200 may also comprise a rendering engine 34 whichcan be configured to provide rendering information that is local to theuser to facilitate operation of the scanners and imaging into the eyesof the user, for the user's view of the world. The rendering engine 34may be implemented by a hardware processor (such as, e.g., a centralprocessing unit or a graphics processing unit). In some embodiments, therendering engine is part of the local processing and data module 260.The rendering engine 34 can be communicatively coupled (e.g., via wiredor wireless links) to other components of the wearable system 200. Forexample, the rendering engine 34, can be coupled to the eye cameras 24via communication link 102, and be coupled to a projecting subsystem 18(which can project light into user's eyes 302, 304 via a scanned laserarrangement in a manner similar to a retinal scanning display) via thecommunication link 104. The rendering engine 34 can also be incommunication with other processing units such as, e.g., the sensor poseprocessor 32 and the image pose processor 36 via links 105 and 94respectively.

The cameras 24 (e.g., mini infrared cameras) may be utilized to trackthe eye pose to support rendering and user input. Some example eye posesmay include where the user is looking or at what depth he or she isfocusing (which may be estimated with eye vergence). The GPS 37, gyros,compass, and accelerometers 39 may be utilized to provide coarse or fastpose estimates. One or more of the cameras 16 can acquire images andpose, which in conjunction with data from an associated cloud computingresource, may be utilized to map the local environment and share userviews with others.

The example components depicted in FIG. 2B are for illustration purposesonly. Multiple sensors and other functional modules are shown togetherfor ease of illustration and description. Some embodiments may includeonly one or a subset of these sensors or modules. Further, the locationsof these components are not limited to the positions depicted in FIG.2B. Some components may be mounted to or housed within other components,such as a belt-mounted component, a hand-held component, or a helmetcomponent. As one example, the image pose processor 36, sensor poseprocessor 32, and rendering engine 34 may be positioned in a beltpackand configured to communicate with other components of the wearablesystem via wireless communication, such as ultra-wideband, Wi-Fi,Bluetooth, etc., or via wired communication. The depicted housing 108preferably is head-mountable and wearable by the user. However, somecomponents of the wearable system 200 may be worn to other portions ofthe user's body. For example, the speaker 240 may be inserted into theears of a user to provide sound to the user.

Regarding the projection of light 38 into the eyes 302, 304 of the user,in some embodiment, the cameras 24 may be utilized to measure where thecenters of a user's eyes 302, 304 are geometrically verged to, which, ingeneral, coincides with a position of focus, or “depth of focus”, of theeyes 302, 304. A 3-dimensional surface of all points the eyes verge tocan be referred to as the “horopter”. The focal distance may take on afinite number of depths, or may be infinitely varying. Light projectedfrom the vergence distance appears to be focused to the subject eye 302,304, while light in front of or behind the vergence distance is blurred.Examples of wearable devices and other display systems of the presentdisclosure are also described in U.S. Patent Publication No.2016/0270656, which is incorporated by reference herein in its entirety.

The human visual system is complicated and providing a realisticperception of depth is challenging. Viewers of an object may perceivethe object as being three-dimensional due to a combination of vergenceand accommodation. Vergence movements (e.g., rolling movements of thepupils toward or away from each other to converge the lines of sight ofthe eyes to fixate upon an object) of the two eyes relative to eachother are closely associated with focusing (or “accommodation”) of thelenses of the eyes. Under normal conditions, changing the focus of thelenses of the eyes, or accommodating the eyes, to change focus from oneobject to another object at a different distance will automaticallycause a matching change in vergence to the same distance, under arelationship known as the “accommodation-vergence reflex.” Likewise, achange in vergence will trigger a matching change in accommodation,under normal conditions. Display systems that provide a better matchbetween accommodation and vergence may form more realistic andcomfortable simulations of three-dimensional imagery.

Further spatially coherent light with a beam diameter of less than about0.7 millimeters can be correctly resolved by the human eye regardless ofwhere the eye focuses. Thus, to create an illusion of proper focaldepth, the eye vergence may be tracked with the cameras 24, and therendering engine 34 and projection subsystem 18 may be utilized torender all objects on or close to the horopter in focus, and all otherobjects at varying degrees of defocus (e.g., using intentionally-createdblurring). Preferably, the system 220 renders to the user at a framerate of about 60 frames per second or greater. As described above,preferably, the cameras 24 may be utilized for eye tracking, andsoftware may be configured to pick up not only vergence geometry butalso focus location cues to serve as user inputs. Preferably, such adisplay system is configured with brightness and contrast suitable forday or night use.

In some embodiments, the display system preferably has latency of lessthan about 20 milliseconds for visual object alignment, less than about0.1 degree of angular alignment, and about 1 arc minute of resolution,which, without being limited by theory, is believed to be approximatelythe limit of the human eye. The display system 220 may be integratedwith a localization system, which may involve GPS elements, opticaltracking, compass, accelerometers, or other data sources, to assist withposition and pose determination; localization information may beutilized to facilitate accurate rendering in the user's view of thepertinent world (e.g., such information would facilitate the glasses toknow where they are with respect to the real world).

In some embodiments, the wearable system 200 is configured to displayone or more virtual images based on the accommodation of the user'seyes. Unlike prior 3D display approaches that force the user to focuswhere the images are being projected, in some embodiments, the wearablesystem is configured to automatically vary the focus of projectedvirtual content to allow for a more comfortable viewing of one or moreimages presented to the user. For example, if the user's eyes have acurrent focus of 1 m, the image may be projected to coincide with theuser's focus. If the user shifts focus to 3 m, the image is projected tocoincide with the new focus. Thus, rather than forcing the user to apredetermined focus, the wearable system 200 of some embodiments allowsthe user's eye to a function in a more natural manner.

Such a wearable system 200 may eliminate or reduce the incidences of eyestrain, headaches, and other physiological symptoms typically observedwith respect to virtual reality devices. To achieve this, variousembodiments of the wearable system 200 are configured to project virtualimages at varying focal distances, through one or more variable focuselements (VFEs). In one or more embodiments, 3D perception may beachieved through a multi-plane focus system that projects images atfixed focal planes away from the user. Other embodiments employ variableplane focus, wherein the focal plane is moved back and forth in thez-direction to coincide with the user's present state of focus.

In both the multi-plane focus systems and variable plane focus systems,wearable system 200 may employ eye tracking to determine a vergence ofthe user's eyes, determine the user's current focus, and project thevirtual image at the determined focus. In other embodiments, wearablesystem 200 comprises a light modulator that variably projects, through afiber scanner, or other light generating source, light beams of varyingfocus in a raster pattern across the retina. Thus, the ability of thedisplay of the wearable system 200 to project images at varying focaldistances not only eases accommodation for the user to view objects in3D, but may also be used to compensate for user ocular anomalies, asfurther described in U.S. Patent Publication No. 2016/0270656, which isincorporated by reference herein in its entirety. In some otherembodiments, a spatial light modulator may project the images to theuser through various optical components. For example, as describedfurther below, the spatial light modulator may project the images ontoone or more waveguides, which then transmit the images to the user.

FIG. 3 illustrates aspects of an approach for simulating athree-dimensional imagery using multiple depth planes. With reference toFIG. 3, objects at various distances from eyes 302 and 304 on the z-axisare accommodated by the eyes 302 and 304 so that those objects are infocus. The eyes 302 and 304 assume particular accommodated states tobring into focus objects at different distances along the z-axis.Consequently, a particular accommodated state may be said to beassociated with a particular one of depth planes 306, which has anassociated focal distance, such that objects or parts of objects in aparticular depth plane are in focus when the eye is in the accommodatedstate for that depth plane. In some embodiments, three-dimensionalimagery may be simulated by providing different presentations of animage for each of the eyes 302 and 304, and also by providing differentpresentations of the image corresponding to each of the depth planes.While shown as being separate for clarity of illustration, it will beappreciated that the fields of view of the eyes 302 and 304 may overlap,for example, as distance along the z-axis increases. In addition, whileshown as flat for the ease of illustration, it will be appreciated thatthe contours of a depth plane may be curved in physical space, such thatall features in a depth plane are in focus with the eye in a particularaccommodated state. Without being limited by theory, it is believed thatthe human eye typically can interpret a finite number of depth planes toprovide depth perception. Consequently, a highly believable simulationof perceived depth may be achieved by providing, to the eye, differentpresentations of an image corresponding to each of these limited numberof depth planes.

Waveguide Stack Assembly

FIG. 4 illustrates an example of a waveguide stack for outputting imageinformation to a user. A wearable system 400 includes a stack ofwaveguides, or stacked waveguide assembly 480 that may be utilized toprovide three-dimensional perception to the eye/brain using a pluralityof waveguides 432 b, 434 b, 436 b, 438 b, 4400 b. In some embodiments,the wearable system 400 may correspond to wearable system 200 of FIG.2A, with FIG. 4 schematically showing some parts of that wearable system200 in greater detail. For example, in some embodiments, the waveguideassembly 480 may be integrated into the display 220 of FIG. 2A.

With continued reference to FIG. 4, the waveguide assembly 480 may alsoinclude a plurality of features 458, 456, 454, 452 between thewaveguides. In some embodiments, the features 458, 456, 454, 452 may belenses. In other embodiments, the features 458, 456, 454, 452 may not belenses. Rather, they may simply be spacers (e.g., cladding layers orstructures for forming air gaps).

The waveguides 432 b, 434 b, 436 b, 438 b, 440 b or the plurality oflenses 458, 456, 454, 452 may be configured to send image information tothe eye with various levels of wavefront curvature or light raydivergence. Each waveguide level may be associated with a particulardepth plane and may be configured to output image informationcorresponding to that depth plane. Image injection devices 420, 422,424, 426, 428 may be utilized to inject image information into thewaveguides 440 b, 438 b, 436 b, 434 b, 432 b, each of which may beconfigured to distribute incoming light across each respectivewaveguide, for output toward the eye 410. Light exits an output surfaceof the image injection devices 420, 422, 424, 426, 428 and is injectedinto a corresponding input edge of the waveguides 440 b, 438 b, 436 b,434 b, 432 b. In some embodiments, a single beam of light (e.g., acollimated beam) may be injected into each waveguide to output an entirefield of cloned collimated beams that are directed toward the eye 410 atparticular angles (and amounts of divergence) corresponding to the depthplane associated with a particular waveguide.

In some embodiments, the image injection devices 420, 422, 424, 426, 428are discrete displays that each produce image information for injectioninto a corresponding waveguide 440 b, 438 b, 436 b, 434 b, 432 b,respectively. In some other embodiments, the image injection devices420, 422, 424, 426, 428 are the output ends of a single multiplexeddisplay which may, e.g., pipe image information via one or more opticalconduits (such as fiber optic cables) to each of the image injectiondevices 420, 422, 424, 426, 428.

A controller 460 controls the operation of the stacked waveguideassembly 480 and the image injection devices 420, 422, 424, 426, 428.The controller 460 includes programming (e.g., instructions in anon-transitory computer-readable medium) that regulates the timing andprovision of image information to the waveguides 440 b, 438 b, 436 b,434 b, 432 b. In some embodiments, the controller 460 may be a singleintegral device, or a distributed system connected by wired or wirelesscommunication channels. The controller 460 may be part of the processingmodules 260 or 270 (illustrated in FIG. 2A) in some embodiments.

The waveguides 440 b, 438 b, 436 b, 434 b, 432 b may be configured topropagate light within each respective waveguide by total internalreflection (TIR). The waveguides 440 b, 438 b, 436 b, 434 b, 432 b mayeach be planar or have another shape (e.g., curved), with major top andbottom surfaces and edges extending between those major top and bottomsurfaces. In the illustrated configuration, the waveguides 440 b, 438 b,436 b, 434 b, 432 b may each include light extracting optical elements440 a, 438 a, 436 a, 434 a, 432 a that are configured to extract lightout of a waveguide by redirecting the light, propagating within eachrespective waveguide, out of the waveguide to output image informationto the eye 410. Extracted light may also be referred to as outcoupledlight, and light extracting optical elements may also be referred to asoutcoupling optical elements. An extracted beam of light is outputted bythe waveguide at locations at which the light propagating in thewaveguide strikes a light redirecting element. The light extractingoptical elements (440 a, 438 a, 436 a, 434 a, 432 a) may, for example,be reflective or diffractive optical features. While illustrateddisposed at the bottom major surfaces of the waveguides 440 b, 438 b,436 b, 434 b, 432 b for ease of description and drawing clarity, in someembodiments, the light extracting optical elements 440 a, 438 a, 436 a,434 a, 432 a may be disposed at the top or bottom major surfaces, or maybe disposed directly in the volume of the waveguides 440 b, 438 b, 436b, 434 b, 432 b. In some embodiments, the light extracting opticalelements 440 a, 438 a, 436 a, 434 a, 432 a may be formed in a layer ofmaterial that is attached to a transparent substrate to form thewaveguides 440 b, 438 b, 436 b, 434 b, 432 b. In some other embodiments,the waveguides 440 b, 438 b, 436 b, 434 b, 432 b may be a monolithicpiece of material and the light extracting optical elements 440 a, 438a, 436 a, 434 a, 432 a may be formed on a surface or in the interior ofthat piece of material.

With continued reference to FIG. 4, as discussed herein, each waveguide440 b, 438 b, 436 b, 434 b, 432 b is configured to output light to forman image corresponding to a particular depth plane. For example, thewaveguide 432 b nearest the eye may be configured to deliver collimatedlight, as injected into such waveguide 432 b, to the eye 410. Thecollimated light may be representative of the optical infinity focalplane. The next waveguide up 434 b may be configured to send outcollimated light which passes through the first lens 452 (e.g., anegative lens) before it can reach the eye 410. First lens 452 may beconfigured to create a slight convex wavefront curvature so that theeye/brain interprets light coming from that next waveguide up 434 b ascoming from a first focal plane closer inward toward the eye 410 fromoptical infinity. Similarly, the third up waveguide 436 b passes itsoutput light through both the first lens 452 and second lens 454 beforereaching the eye 410. The combined optical power of the first and secondlenses 452 and 454 may be configured to create another incrementalamount of wavefront curvature so that the eye/brain interprets lightcoming from the third waveguide 436 b as coming from a second focalplane that is even closer inward toward the person from optical infinitythan was light from the next waveguide up 434 b.

The other waveguide layers (e.g., waveguides 438 b, 440 b) and lenses(e.g., lenses 456, 458) are similarly configured, with the highestwaveguide 440 b in the stack sending its output through all of thelenses between it and the eye for an aggregate focal powerrepresentative of the closest focal plane to the person. To compensatefor the stack of lenses 458, 456, 454, 452 when viewing/interpretinglight coming from the world 470 on the other side of the stackedwaveguide assembly 480, a compensating lens layer 430 may be disposed atthe top of the stack to compensate for the aggregate power of the lensstack 458, 456, 454, 452 below. Such a configuration provides as manyperceived focal planes as there are available waveguide/lens pairings.Both the light extracting optical elements of the waveguides and thefocusing aspects of the lenses may be static (e.g., not dynamic orelectro-active). In some alternative embodiments, either or both may bedynamic using electro-active features.

With continued reference to FIG. 4, the light extracting opticalelements 440 a, 438 a, 436 a, 434 a, 432 a may be configured to bothredirect light out of their respective waveguides and to output thislight with the appropriate amount of divergence or collimation for aparticular depth plane associated with the waveguide. As a result,waveguides having different associated depth planes may have differentconfigurations of light extracting optical elements, which output lightwith a different amount of divergence depending on the associated depthplane. In some embodiments, as discussed herein, the light extractingoptical elements 440 a, 438 a, 436 a, 434 a, 432 a may be volumetric orsurface features, which may be configured to output light at specificangles. For example, the light extracting optical elements 440 a, 438 a,436 a, 434 a, 432 a may be volume holograms, surface holograms, ordiffraction gratings. Light extracting optical elements, such asdiffraction gratings, are described in U.S. Patent Publication No.2015/0178939, published Jun. 25, 2015, which is incorporated byreference herein in its entirety.

In some embodiments, the light extracting optical elements 440 a, 438 a,436 a, 434 a, 432 a are diffractive features that form a diffractionpattern, or “diffractive optical element” (also referred to herein as a“DOE”). Preferably, the DOE has a relatively low diffraction efficiencyso that only a portion of the light of the beam is deflected away towardthe eye 410 with each intersection of the DOE, while the rest continuesto move through a waveguide via total internal reflection. The lightcarrying the image information can thus be divided into a number ofrelated exit beams that exit the waveguide at a multiplicity oflocations and the result is a fairly uniform pattern of exit emissiontoward the eye 304 for this particular collimated beam bouncing aroundwithin a waveguide.

In some embodiments, one or more DOEs may be switchable between “on”state in which they actively diffract, and “off” state in which they donot significantly diffract. For instance, a switchable DOE may comprisea layer of polymer dispersed liquid crystal, in which microdropletscomprise a diffraction pattern in a host medium, and the refractiveindex of the microdroplets can be switched to substantially match therefractive index of the host material (in which case the pattern doesnot appreciably diffract incident light) or the microdroplet can beswitched to an index that does not match that of the host medium (inwhich case the pattern actively diffracts incident light).

In some embodiments, the number and distribution of depth planes ordepth of field may be varied dynamically based on the pupil sizes ororientations of the eyes of the viewer. Depth of field may changeinversely with a viewer's pupil size. As a result, as the sizes of thepupils of the viewer's eyes decrease, the depth of field increases suchthat one plane that is not discernible because the location of thatplane is beyond the depth of focus of the eye may become discernible andappear more in focus with reduction of pupil size and commensurate withthe increase in depth of field. Likewise, the number of spaced apartdepth planes used to present different images to the viewer may bedecreased with the decreased pupil size. For example, a viewer may notbe able to clearly perceive the details of both a first depth plane anda second depth plane at one pupil size without adjusting theaccommodation of the eye away from one depth plane and to the otherdepth plane. These two depth planes may, however, be sufficiently infocus at the same time to the user at another pupil size withoutchanging accommodation.

In some embodiments, the display system may vary the number ofwaveguides receiving image information based upon determinations ofpupil size or orientation, or upon receiving electrical signalsindicative of particular pupil size or orientation. For example, if theuser's eyes are unable to distinguish between two depth planesassociated with two waveguides, then the controller 460 (which may be anembodiment of the local processing and data module 260) can beconfigured or programmed to cease providing image information to one ofthese waveguides. Advantageously, this may reduce the processing burdenon the system, thereby increasing the responsiveness of the system. Inembodiments in which the DOEs for a waveguide are switchable between theon and off states, the DOEs may be switched to the off state when thewaveguide does receive image information.

In some embodiments, it may be desirable to have an exit beam meet thecondition of having a diameter that is less than the diameter of the eyeof a viewer. However, meeting this condition may be challenging in viewof the variability in size of the viewer's pupils. In some embodiments,this condition is met over a wide range of pupil sizes by varying thesize of the exit beam in response to determinations of the size of theviewer's pupil. For example, as the pupil size decreases, the size ofthe exit beam may also decrease. In some embodiments, the exit beam sizemay be varied using a variable aperture.

The wearable system 400 can include an outward-facing imaging system 464(e.g., a digital camera) that images a portion of the world 470. Thisportion of the world 470 may be referred to as the field of view (FOV)of a world camera and the imaging system 464 is sometimes referred to asan FOV camera. The FOV of the world camera may or may not be the same asthe FOV of a viewer 210 which encompasses a portion of the world 470 theviewer 210 perceives at a given time. For example, in some situations,the FOV of the world camera may be larger than the viewer 210 of theviewer 210 of the wearable system 400. The entire region available forviewing or imaging by a viewer may be referred to as the field of regard(FOR). The FOR may include 4π steradians of solid angle surrounding thewearable system 400 because the wearer can move his body, head, or eyesto perceive substantially any direction in space. In other contexts, thewearer's movements may be more constricted, and accordingly the wearer'sFOR may subtend a smaller solid angle. Images obtained from theoutward-facing imaging system 464 can be used to track gestures made bythe user (e.g., hand or finger gestures), detect objects in the world470 in front of the user, and so forth.

The wearable system 400 can include an audio sensor 232, e.g., amicrophone, to capture ambient sound. As described above, in someembodiments, one or more other audio sensors can be positioned toprovide stereo sound reception useful to the determination of locationof a speech source. The audio sensor 232 can comprise a directionalmicrophone, as another example, which can also provide such usefuldirectional information as to where the audio source is located.

The wearable system 400 can also include an inward-facing imaging system466 (e.g., a digital camera), which observes the movements of the user,such as the eye movements and the facial movements. The inward-facingimaging system 466 may be used to capture images of the eye 410 todetermine the size or orientation of the pupil of the eye 304. Theinward-facing imaging system 466 can be used to obtain images for use indetermining the direction the user is looking (e.g., eye pose) or forbiometric identification of the user (e.g., via iris identification). Insome embodiments, at least one camera may be utilized for each eye, toseparately determine the pupil size or eye pose of each eyeindependently, thereby allowing the presentation of image information toeach eye to be dynamically tailored to that eye. In some otherembodiments, the pupil diameter or orientation of only a single eye 410(e.g., using only a single camera per pair of eyes) is determined andassumed to be similar for both eyes of the user. The images obtained bythe inward-facing imaging system 466 may be analyzed to determine theuser's eye pose or mood, which can be used by the wearable system 400 todecide which audio or visual content should be presented to the user.The wearable system 400 may also determine head pose (e.g., headposition or head orientation) using sensors such as IMUs,accelerometers, gyroscopes, etc.

The wearable system 400 can include a user input device 466 by which theuser can input commands to the controller 460 to interact with thewearable system 400. For example, the user input device 466 can includea trackpad, a touchscreen, a joystick, a multiple degree-of-freedom(DOF) controller, a capacitive sensing device, a game controller, akeyboard, a mouse, a directional pad (D-pad), a wand, a haptic device, atotem (e.g., functioning as a virtual user input device), and so forth.A multi-DOF controller can sense user input in some or all possibletranslations (e.g., left/right, forward/backward, or up/down) orrotations (e.g., yaw, pitch, or roll) of the controller. A multi-DOFcontroller which supports the translation movements may be referred toas a 3DOF while a multi-DOF controller which supports the translationsand rotations may be referred to as 6DOF. In some cases, the user mayuse a finger (e.g., a thumb) to press or swipe on a touch-sensitiveinput device to provide input to the wearable system 400 (e.g., toprovide user input to a user interface provided by the wearable system400). The user input device 466 may be held by the user's hand duringthe use of the wearable system 400. The user input device 466 can be inwired or wireless communication with the wearable system 400.

FIG. 5 shows an example of exit beams outputted by a waveguide. Onewaveguide is illustrated, but it will be appreciated that otherwaveguides in the waveguide assembly 480 may function similarly, wherethe waveguide assembly 480 includes multiple waveguides. Light 520 isinjected into the waveguide 432 b at the input edge 432 c of thewaveguide 432 b and propagates within the waveguide 432 b by TIR. Atpoints where the light 520 impinges on the DOE 432 a, a portion of thelight exits the waveguide as exit beams 510. The exit beams 510 areillustrated as substantially parallel but they may also be redirected topropagate to the eye 410 at an angle (e.g., forming divergent exitbeams), depending on the depth plane associated with the waveguide 432b. It will be appreciated that substantially parallel exit beams may beindicative of a waveguide with light extracting optical elements thatoutcouple light to form images that appear to be set on a depth plane ata large distance (e.g., optical infinity) from the eye 410. Otherwaveguides or other sets of light extracting optical elements may outputan exit beam pattern that is more divergent, which would require the eye410 to accommodate to a closer distance to bring it into focus on theretina and would be interpreted by the brain as light from a distancecloser to the eye 410 than optical infinity.

FIG. 6 is a schematic diagram showing an optical system including awaveguide apparatus, an optical coupler subsystem to optically couplelight to or from the waveguide apparatus, and a control subsystem, usedin the generation of a multi-focal volumetric display, image, or lightfield. The optical system can include a waveguide apparatus, an opticalcoupler subsystem to optically couple light to or from the waveguideapparatus, and a control subsystem. The optical system can be used togenerate a multi-focal volumetric, image, or light field. The opticalsystem can include one or more primary planar waveguides 632 a (only oneis shown in FIG. 6) and one or more DOEs 632 b associated with each ofat least some of the primary waveguides 632 a. The planar waveguides 632b can be similar to the waveguides 432 b, 434 b, 436 b, 438 b, 440 bdiscussed with reference to FIG. 4. The optical system may employ adistribution waveguide apparatus to relay light along a first axis(vertical or Y-axis in view of FIG. 6), and expand the light's effectiveexit pupil along the first axis (e.g., Y-axis). The distributionwaveguide apparatus may, for example, include a distribution planarwaveguide 622 b and at least one DOE 622 a (illustrated by doubledash-dot line) associated with the distribution planar waveguide 622 b.The distribution planar waveguide 622 b may be similar or identical inat least some respects to the primary planar waveguide 632 b, having adifferent orientation therefrom. Likewise, at least one DOE 622 a may besimilar to or identical in at least some respects to the DOE 632 a. Forexample, the distribution planar waveguide 622 b or DOE 622 a may becomprised of the same materials as the primary planar waveguide 632 b orDOE 632 a, respectively. Embodiments of the optical display system 600shown in FIG. 6 can be integrated into the wearable system 200 shown inFIG. 2A.

The relayed and exit-pupil expanded light may be optically coupled fromthe distribution waveguide apparatus into the one or more primary planarwaveguides 632 b. The primary planar waveguide 632 b can relay lightalong a second axis, preferably orthogonal to first axis (e.g.,horizontal or X-axis in view of FIG. 6). Notably, the second axis can bea non-orthogonal axis to the first axis. The primary planar waveguide632 b expands the light's effective exit pupil along that second axis(e.g., X-axis). For example, the distribution planar waveguide 622 b canrelay and expand light along the vertical or Y-axis, and pass that lightto the primary planar waveguide 632 b which can relay and expand lightalong the horizontal or X-axis.

The optical system may include one or more sources of colored light(e.g., red, green, and blue laser light) 610 which may be opticallycoupled into a proximal end of a single mode optical fiber 640. A distalend of the optical fiber 640 may be threaded or received through ahollow tube 642 of piezoelectric material. The distal end protrudes fromthe tube 642 as fixed-free flexible cantilever 644. The piezoelectrictube 642 can be associated with four quadrant electrodes (notillustrated). The electrodes may, for example, be plated on the outside,outer surface or outer periphery or diameter of the tube 642. A coreelectrode (not illustrated) may also be located in a core, center, innerperiphery or inner diameter of the tube 642.

Drive electronics 650, for example electrically coupled via wires 660,drive opposing pairs of electrodes to bend the piezoelectric tube 642 intwo axes independently. The protruding distal tip of the optical fiber644 has mechanical modes of resonance. The frequencies of resonance candepend upon a diameter, length, and material properties of the opticalfiber 644. By vibrating the piezoelectric tube 642 near a first mode ofmechanical resonance of the fiber cantilever 644, the fiber cantilever644 can be caused to vibrate, and can sweep through large deflections.

By stimulating resonant vibration in two axes, the tip of the fibercantilever 644 is scanned biaxially in an area filling two-dimensional(2D) scan. By modulating an intensity of light source(s) 610 insynchrony with the scan of the fiber cantilever 644, light emerging fromthe fiber cantilever 644 can form an image. Descriptions of such a setup are provided in U.S. Patent Publication No. 2014/0003762, which isincorporated by reference herein in its entirety.

A component of an optical coupler subsystem can collimate the lightemerging from the scanning fiber cantilever 644. The collimated lightcan be reflected by mirrored surface 648 into the narrow distributionplanar waveguide 622 b which contains the at least one diffractiveoptical element (DOE) 622 a. The collimated light can propagatevertically (relative to the view of FIG. 6) along the distributionplanar waveguide 622 b by TIR, and in doing so repeatedly intersectswith the DOE 622 a. The DOE 622 a preferably has a low diffractionefficiency. This can cause a fraction (e.g., 10%) of the light to bediffracted toward an edge of the larger primary planar waveguide 632 bat each point of intersection with the DOE 622 a, and a fraction of thelight to continue on its original trajectory down the length of thedistribution planar waveguide 622 b via TIR.

At each point of intersection with the DOE 622 a, additional light canbe diffracted toward the entrance of the primary waveguide 632 b. Bydividing the incoming light into multiple outcoupled sets, the exitpupil of the light can be expanded vertically by the DOE 622 a in thedistribution planar waveguide 622 b. This vertically expanded lightcoupled out of distribution planar waveguide 622 b can enter the edge ofthe primary planar waveguide 632 b.

Light entering primary waveguide 632 b can propagate horizontally(relative to the view of FIG. 6) along the primary waveguide 632 b viaTIR. As the light intersects with DOE 632 a at multiple points as itpropagates horizontally along at least a portion of the length of theprimary waveguide 632 b via TIR. The DOE 632 a may advantageously bedesigned or configured to have a phase profile that is a summation of alinear diffraction pattern and a radially symmetric diffractive pattern,to produce both deflection and focusing of the light. The DOE 632 a mayadvantageously have a low diffraction efficiency (e.g., 10%), so thatonly a portion of the light of the beam is deflected toward the eye ofthe view with each intersection of the DOE 632 a while the rest of thelight continues to propagate through the primary waveguide 632 b viaTIR.

At each point of intersection between the propagating light and the DOE632 a, a fraction of the light is diffracted toward the adjacent face ofthe primary waveguide 632 b allowing the light to escape the TIR, andemerge from the face of the primary waveguide 632 b. In someembodiments, the radially symmetric diffraction pattern of the DOE 632 aadditionally imparts a focus level to the diffracted light, both shapingthe light wavefront (e.g., imparting a curvature) of the individual beamas well as steering the beam at an angle that matches the designed focuslevel.

Accordingly, these different pathways can cause the light to be coupledout of the primary planar waveguide 632 b by a multiplicity of DOEs 632a at different angles, focus levels, or yielding different fill patternsat the exit pupil. Different fill patterns at the exit pupil can bebeneficially used to create a light field display with multiple depthplanes. Each layer in the waveguide assembly or a set of layers (e.g., 3layers) in the stack may be employed to generate a respective color(e.g., red, blue, green). Thus, for example, a first set of threeadjacent layers may be employed to respectively produce red, blue andgreen light at a first focal depth. A second set of three adjacentlayers may be employed to respectively produce red, blue and green lightat a second focal depth. Multiple sets may be employed to generate afull 3D or 4D color image light field with various focal depths.

Other Components of the Wearable System

In many implementations, the wearable system may include othercomponents in addition or in alternative to the components of thewearable system described above. The wearable system may, for example,include one or more haptic devices or components. The haptic devices orcomponents may be operable to provide a tactile sensation to a user. Forexample, the haptic devices or components may provide a tactilesensation of pressure or texture when touching virtual content (e.g.,virtual objects, virtual tools, other virtual constructs). The tactilesensation may replicate a feel of a physical object which a virtualobject represents, or may replicate a feel of an imagined object orcharacter (e.g., a dragon) which the virtual content represents. In someimplementations, haptic devices or components may be worn by the user(e.g., a user wearable glove). In some implementations, haptic devicesor components may be held by the user.

The wearable system may, for example, include one or more physicalobjects which are manipulable by the user to allow input or interactionwith the wearable system. These physical objects may be referred toherein as totems. Some totems may take the form of inanimate objects,such as for example, a piece of metal or plastic, a wall, a surface oftable. In certain implementations, the totems may not actually have anyphysical input structures (e.g., keys, triggers, joystick, trackball,rocker switch). Instead, the totem may simply provide a physicalsurface, and the wearable system may render a user interface so as toappear to a user to be on one or more surfaces of the totem. Forexample, the wearable system may render an image of a computer keyboardand trackpad to appear to reside on one or more surfaces of a totem. Forexample, the wearable system may render a virtual computer keyboard andvirtual trackpad to appear on a surface of a thin rectangular plate ofaluminum which serves as a totem. The rectangular plate does not itselfhave any physical keys or trackpad or sensors. However, the wearablesystem may detect user manipulation or interaction or touches with therectangular plate as selections or inputs made via the virtual keyboardor virtual trackpad. The user input device 466 (shown in FIG. 4) may bean embodiment of a totem, which may include a trackpad, a touchpad, atrigger, a joystick, a trackball, a rocker or virtual switch, a mouse, akeyboard, a multi-degree-of-freedom controller, or another physicalinput device. A user may use the totem, alone or in combination withposes, to interact with the wearable system or other users.

Examples of haptic devices and totems usable with the wearable devices,HMD, and display systems of the present disclosure are described in U.S.Patent Publication No. 2015/0016777, which is incorporated by referenceherein in its entirety.

Example Wearable Systems, Environments, and Interfaces

A wearable system may employ various mapping related techniques in orderto achieve high depth of field in the rendered light fields. In mappingout the virtual world, it is advantageous to know all the features andpoints in the real world to accurately portray virtual objects inrelation to the real world. To this end, FOV images captured from usersof the wearable system can be added to a world model by including newpictures that convey information about various points and features ofthe real world. For example, the wearable system can collect a set ofmap points (such as 2D points or 3D points) and find new map points torender a more accurate version of the world model. The world model of afirst user can be communicated (e.g., over a network such as a cloudnetwork) to a second user so that the second user can experience theworld surrounding the first user.

FIG. 7 is a block diagram of an example of an MR environment 700. The MRenvironment 700 may be configured to receive input (e.g., visual input702 from the user's wearable system, stationary input 704 such as roomcameras, sensory input 706 from various sensors, gestures, totems, eyetracking, user input from the user input device 466 etc.) from one ormore user wearable systems (e.g., wearable system 200 or display system220) or stationary room systems (e.g., room cameras, etc.). The wearablesystems can use various sensors (e.g., accelerometers, gyroscopes,temperature sensors, movement sensors, depth sensors, GPS sensors,inward-facing imaging system, outward-facing imaging system, etc.) todetermine the location and various other attributes of the environmentof the user. This information may further be supplemented withinformation from stationary cameras in the room that may provide imagesor various cues from a different point of view. The image data acquiredby the cameras (such as the room cameras or the cameras of theoutward-facing imaging system) may be reduced to a set of mappingpoints.

One or more object recognizers 708 can crawl through the received data(e.g., the collection of points) and recognize or map points, tagimages, attach semantic information to objects with the help of a mapdatabase 710. The map database 710 may comprise various points collectedover time and their corresponding objects. The various devices and themap database can be connected to each other through a network (e.g.,LAN, WAN, etc.) to access the cloud.

Based on this information and collection of points in the map database,the object recognizers 708 a to 708 n may recognize objects in anenvironment. For example, the object recognizers can recognize faces,persons, windows, walls, user input devices, televisions, documents(e.g., travel tickets, driver's license, passport as described in thesecurity examples herein), other objects in the user's environment, etc.One or more object recognizers may be specialized for object withcertain characteristics. For example, the object recognizer 708 a may beused to recognizer faces, while another object recognizer may be usedrecognize documents.

The object recognitions may be performed using a variety of computervision techniques. For example, the wearable system can analyze theimages acquired by the outward-facing imaging system 464 (shown in FIG.4) to perform scene reconstruction, event detection, video tracking,object recognition (e.g., persons or documents), object pose estimation,facial recognition (e.g., from a person in the environment or an imageon a document), learning, indexing, motion estimation, or image analysis(e.g., identifying indicia within documents such as photos, signatures,identification information, travel information, etc.), and so forth. Oneor more computer vision algorithms may be used to perform these tasks.Non-limiting examples of computer vision algorithms include:Scale-invariant feature transform (SIFT), speeded up robust features(SURF), oriented FAST and rotated BRIEF (ORB), binary robust invariantscalable keypoints (BRISK), fast retina keypoint (FREAK), Viola-Jonesalgorithm, Eigenfaces approach, Lucas-Kanade algorithm, Horn-Schunkalgorithm, Mean-shift algorithm, visual simultaneous location andmapping (vSLAM) techniques, a sequential Bayesian estimator (e.g.,Kalman filter, extended Kalman filter, etc.), bundle adjustment,Adaptive thresholding (and other thresholding techniques), IterativeClosest Point (ICP), Semi Global Matching (SGM), Semi Global BlockMatching (SGBM), Feature Point Histograms, various machine learningalgorithms (such as e.g., support vector machine, k-nearest neighborsalgorithm, Naive Bayes, neural network (including convolutional or deepneural networks), or other supervised/unsupervised models, etc.), and soforth.

One or more object recognizers 708 can also implement various textrecognition algorithms to identify and extract the text from the images.Some example text recognition algorithms include: optical characterrecognition (OCR) algorithms, deep learning algorithms (such as deepneural networks), pattern matching algorithms, algorithms forpre-processing, etc.

The object recognitions can additionally or alternatively be performedby a variety of machine learning algorithms. Once trained, the machinelearning algorithm can be stored by the HMD. Some examples of machinelearning algorithms can include supervised or non-supervised machinelearning algorithms, including regression algorithms (such as, forexample, Ordinary Least Squares Regression), instance-based algorithms(such as, for example, Learning Vector Quantization), decision treealgorithms (such as, for example, classification and regression trees),Bayesian algorithms (such as, for example, Naive Bayes), clusteringalgorithms (such as, for example, k-means clustering), association rulelearning algorithms (such as, for example, a-priori algorithms),artificial neural network algorithms (such as, for example, Perceptron),deep learning algorithms (such as, for example, Deep Boltzmann Machine,or deep neural network), dimensionality reduction algorithms (such as,for example, Principal Component Analysis), ensemble algorithms (suchas, for example, Stacked Generalization), or other machine learningalgorithms. In some embodiments, individual models can be customized forindividual data sets. For example, the wearable device can generate orstore a base model. The base model may be used as a starting point togenerate additional models specific to a data type (e.g., a particularuser in the telepresence session), a data set (e.g., a set of additionalimages obtained of the user in the telepresence session), conditionalsituations, or other variations. In some embodiments, the wearable HMDcan be configured to utilize a plurality of techniques to generatemodels for analysis of the aggregated data. Other techniques may includeusing pre-defined thresholds or data values.

Based on this information and collection of points in the map database,the object recognizers 708 a to 708 n may recognize objects andsupplement objects with semantic information to give life to theobjects. For example, if the object recognizer recognizes a set ofpoints to be a door, the system may attach some semantic information(e.g., the door has a hinge and has a 90 degree movement about thehinge). If the object recognizer recognizes a set of points to be amirror, the system may attach semantic information that the mirror has areflective surface that can reflect images of objects in the room. Thesemantic information can include affordances of the objects as describedherein. For example, the semantic information may include a normal ofthe object. The system can assign a vector whose direction indicates thenormal of the object. Over time the map database grows as the system(which may reside locally or may be accessible through a wirelessnetwork) accumulates more data from the world. Once the objects arerecognized, the information may be transmitted to one or more wearablesystems. For example, the MR environment 700 may include informationabout a scene happening in California. The environment 700 may betransmitted to one or more users in New York. Based on data receivedfrom an FOV camera and other inputs, the object recognizers and othersoftware components can map the points collected from the variousimages, recognize objects etc., such that the scene may be accurately“passed over” to a second user, who may be in a different part of theworld. The environment 700 may also use a topological map forlocalization purposes.

FIG. 8 is a process flow diagram of an example of a method 800 ofrendering virtual content in relation to recognized objects. The method800 describes how a virtual scene may be presented to a user of thewearable system. The user may be geographically remote from the scene.For example, the user may be in New York, but may want to view a scenethat is presently going on in California, or may want to go on a walkwith a friend who resides in California.

At block 810, the wearable system may receive input from the user andother users regarding the environment of the user. This may be achievedthrough various input devices, and knowledge already possessed in themap database. The user's FOV camera, sensors, GPS, eye tracking, etc.,convey information to the system at block 810. The system may determinesparse points based on this information at block 820. The sparse pointsmay be used in determining pose data (e.g., head pose, eye pose, bodypose, or hand gestures) that can be used in displaying and understandingthe orientation and position of various objects in the user'ssurroundings. The object recognizers 708 a-708 n may crawl through thesecollected points and recognize one or more objects using a map databaseat block 830. This information may then be conveyed to the user'sindividual wearable system at block 840, and the desired virtual scenemay be accordingly displayed to the user at block 850. For example, thedesired virtual scene (e.g., user in CA) may be displayed at theappropriate orientation, position, etc., in relation to the variousobjects and other surroundings of the user in New York.

FIG. 9 is a block diagram of another example of a wearable system. Inthis example, the wearable system 900 comprises a map 920, which mayinclude the map database 710 containing map data for the world. The mapmay partly reside locally on the wearable system, and may partly resideat networked storage locations accessible by wired or wireless network(e.g., in a cloud system). A pose process 910 may be executed on thewearable computing architecture (e.g., processing module 260 orcontroller 460) and utilize data from the map 920 to determine positionand orientation of the wearable computing hardware or user. Pose datamay be computed from data collected on the fly as the user isexperiencing the system and operating in the world. The data maycomprise images, data from sensors (such as inertial measurement units,which generally comprise accelerometer and gyroscope components) andsurface information pertinent to objects in the real or virtualenvironment.

A sparse point representation may be the output of a simultaneouslocalization and mapping (e.g., SLAM or vSLAM, referring to aconfiguration wherein the input is images/visual only) process. Thesystem can be configured to not only find out where in the world thevarious components are, but what the world is made of. Pose may be abuilding block that achieves many goals, including populating the mapand using the data from the map.

In one embodiment, a sparse point position may not be completelyadequate on its own, and further information may be needed to produce amultifocal AR, VR, or MR experience. Dense representations, generallyreferring to depth map information, may be utilized to fill this gap atleast in part. Such information may be computed from a process referredto as Stereo 940, wherein depth information is determined using atechnique such as triangulation or time-of-flight sensing. Imageinformation and active patterns (such as infrared patterns created usingactive projectors), images acquired from image cameras, or handgestures/totem 950 may serve as input to the Stereo process 940. Asignificant amount of depth map information may be fused together, andsome of this may be summarized with a surface representation. Forexample, mathematically definable surfaces may be efficient (e.g.,relative to a large point cloud) and digestible inputs to otherprocessing devices like game engines. Thus, the output of the stereoprocess (e.g., a depth map) 940 may be combined in the fusion process930. Pose 910 may be an input to this fusion process 930 as well, andthe output of fusion 930 becomes an input to populating the map process920. Sub-surfaces may connect with each other, such as in topographicalmapping, to form larger surfaces, and the map becomes a large hybrid ofpoints and surfaces.

To resolve various aspects in a mixed reality process 960, variousinputs may be utilized. For example, in the embodiment depicted in FIG.9, Game parameters may be inputs to determine that the user of thesystem is playing a monster battling game with one or more monsters atvarious locations, monsters dying or running away under variousconditions (such as if the user shoots the monster), walls or otherobjects at various locations, and the like. The world map may includeinformation regarding the location of the objects or semanticinformation of the objects and the world map can be another valuableinput to mixed reality. Pose relative to the world becomes an input aswell and plays a key role to almost any interactive system.

Controls or inputs from the user are another input to the wearablesystem 900. As described herein, user inputs can include visual input,gestures, totems, audio input, sensory input, etc. In order to movearound or play a game, for example, the user may need to instruct thewearable system 900 regarding what he or she wants to do. Beyond justmoving oneself in space, there are various forms of user controls thatmay be utilized. In one embodiment, a totem (e.g. a user input device),or an object such as a toy gun may be held by the user and tracked bythe system. The system preferably will be configured to know that theuser is holding the item and understand what kind of interaction theuser is having with the item (e.g., if the totem or object is a gun, thesystem may be configured to understand location and orientation, as wellas whether the user is clicking a trigger or other sensed button orelement which may be equipped with a sensor, such as an IMU, which mayassist in determining what is going on, even when such activity is notwithin the field of view of any of the cameras.)

Hand gesture tracking or recognition may also provide input information.The wearable system 900 may be configured to track and interpret handgestures for button presses, for gesturing left or right, stop, grab,hold, etc. For example, in one configuration, the user may want to flipthrough emails or a calendar in a non-gaming environment, or do a “fistbump” with another person or player. The wearable system 900 may beconfigured to leverage a minimum amount of hand gesture, which may ormay not be dynamic. For example, the gestures may be simple staticgestures like open hand for stop, thumbs up for ok, thumbs down for notok; or a hand flip right, or left, or up/down for directional commands.Hand gesture tracking can include tracking gestures made by others inthe user's environment, such as others who make the gestures tocommunicate with sign language (see, e.g., FIG. 13A).

Eye tracking is another input (e.g., tracking where the user is lookingto control the display technology to render at a specific depth orrange). In one embodiment, vergence of the eyes may be determined usingtriangulation, and then using a vergence/accommodation model developedfor that particular person, accommodation may be determined. Eyetracking can be performed by the eye camera(s) to determine eye gaze(e.g., direction or orientation of one or both eyes). Other techniquescan be used for eye tracking such as, e.g., measurement of electricalpotentials by electrodes placed near the eye(s) (e.g.,electrooculography).

Speech tracking can be another input can be used alone or in combinationwith other inputs (e.g., totem tracking, eye tracking, gesture tracking,etc.). Speech tracking may include speech recognition, voicerecognition, alone or in combination. The system 900 can include anaudio sensor (e.g., a microphone) that receives an audio stream from theenvironment. The system 900 can incorporate voice recognition technologyto determine who is speaking (e.g., whether the speech is from thewearer of the ARD or another person or voice (e.g., a recorded voicetransmitted by a loudspeaker in the environment)) as well as speechrecognition technology to determine what is being said. The local data &processing module 260 or the remote processing module 270 can processthe audio data from the microphone (or audio data in another stream suchas, e.g., a video stream being watched by the user) to identify contentof the speech by applying various speech recognition algorithms, suchas, e.g., hidden Markov models, dynamic time warping (DTW)-based speechrecognitions, neural networks, deep learning algorithms such as deepfeedforward and recurrent neural networks, end-to-end automatic speechrecognitions, machine learning algorithms (described with reference toFIG. 7), or other algorithms that uses acoustic modeling or languagemodeling, etc.

Another input to the mixed reality process 960 can include trackingsignage in the environment. Signage can include commercial or publicdisplay signs. As described with reference to FIGS. 16A-19, the systemcan recognize signage, identify text in the signage, adjustcharacteristics of the text (e.g., increasing a font size of the text toimprove readability), modify the content of the text (e.g., translatethe text from a foreign language to a language understood by the user),etc.

The local data & processing module 260 or the remote processing module270 can also apply voice recognition algorithms which can identify theidentity of the speaker, such as whether the speaker is the user 210 ofthe wearable system 900 or another person with whom the user isconversing. Some example voice recognition algorithms can includefrequency estimation, hidden Markov models, Gaussian mixture models,pattern matching algorithms, neural networks, matrix representation,Vector Quantization, speaker diarisation, decision trees, and dynamictime warping (DTW) technique. Voice recognition techniques can alsoinclude anti-speaker techniques, such as cohort models, and worldmodels. Spectral features may be used in representing speakercharacteristics. The local data & processing module or the remote dataprocessing module 270 can use various machine learning algorithmsdescribed with reference to FIG. 7 to perform the voice recognition.

The system 900 can also include a sensory eyewear system 970 forfacilitating a user's interactions with other people or the environment.An implementation of a sensory eyewear system 970 can use these usercontrols or inputs via a UI. UI elements (e.g., controls, popup windows,bubbles, data entry fields, etc.) can be used, for example, to dismiss adisplay of information, e.g., converted text, graphics, or auxiliaryinformation, or to request a display of auxiliary information. The UIcan permit the user to input a list of one or more languages the userunderstands so that the sensory eyewear system 970 knows which languageto use in translating signs made by a conversation partner in a signlanguage. Examples of such implementations and these uses are describedfurther below.

The sensory eyewear system 970 can also comprise text recognition,modification, and rendering features. Such features may in combinationwith various other components of the wearable system to enhance a user'sinteractions with the environment. For example, an HMD can includes oneor more light sources 11 that are configured to project an image ontothe display based on text that is identified from an image of the user'sphysical environment (e.g., such that the projected image occludes theoriginal text from the physical environment). An optically transmissiveeyepiece 106 can be configured to transmit light from the one or morelight sources 11 to the user 210 as an image. The image may appear as ifit is at a particular depth, which may be just one of many possibledepths at which the HMD system 200 could have displayed the image. TheHMD system 100 may be able to project images to appear at a number ofdifferent depths, which may appear as if on different depth planes 306(see FIG. 3). In some embodiments where the eyepiece 106 is opticallytransmissive, the eyepiece 106 can allow light from the environment toenter a user's eye. Thus, in such embodiments, a user 210 may seeportions of an image from the environment together with projected imagesfrom the one or more light sources 11.

With regard to the camera systems, the example wearable system 900 shownin FIG. 9 can include three pairs of cameras: a relative wide FOV orpassive SLAM pair of cameras arranged to the sides of the user's face, adifferent pair of cameras oriented in front of the user to handle thestereo imaging process 940 and also to capture hand gestures andtotem/object tracking in front of the user's face. The FOV cameras orthe pair of cameras for stereo process 940 may also be referred to ascameras 16. The FOV cameras and the pair of cameras for the stereoprocess 940 may be a part of the outward-facing imaging system 464(shown in FIG. 4). The wearable system 900 can include eye trackingcameras (which also were shown as eye cameras 24 and which may be a partof an inward-facing imaging system 462 shown in FIG. 4) oriented towardthe eyes of the user in order to triangulate eye vectors and otherinformation. The wearable system 900 may also comprise one or moretextured light projectors (such as infrared (IR) projectors) to injecttexture into a scene.

FIG. 10 is a process flow diagram of an example of a method 1000 fordetermining user input to a wearable system. In this example, the usermay interact with a totem. The user may have multiple totems. Forexample, the user may have designated one totem for a social mediaapplication, another totem for playing games, etc. At block 1010, thewearable system may detect a motion of a totem. The movement of thetotem may be recognized through the outward-facing imaging system or maybe detected through sensors (e.g., haptic glove, image sensors, handtracking devices, eye-tracking cameras, head pose sensors, etc.).

Based at least partly on the detected gesture, eye pose, head pose, orinput through the totem, the wearable system detects a position,orientation, or movement of the totem (or the user's eyes or head orgestures) with respect to a reference frame, at block 1020. Thereference frame may be a set of map points based on which the wearablesystem translates the movement of the totem (or the user) to an actionor command. At block 1030, the user's interaction with the totem ismapped. Based on the mapping of the user interaction with respect to thereference frame 1020, the system determines the user input at block1040.

For example, the user may move a totem or physical object back and forthto signify turning a virtual page and moving on to a next page or movingfrom one user interface (UI) display screen to another UI screen. Asanother example, the user may move their head or eyes to look atdifferent real or virtual objects in the user's FOR. If the user's gazeat a particular real or virtual object is longer than a threshold time,the real or virtual object may be selected as the user input. In someimplementations, the vergence of the user's eyes can be tracked and anaccommodation/vergence model can be used to determine the accommodationstate of the user's eyes, which provides information on a depth plane onwhich the user is focusing. In some implementations, the wearable systemcan use ray casting techniques to determine which real or virtualobjects are along the direction of the user's head pose or eye pose. Invarious implementations, the ray casting techniques can include castingthin, pencil rays with substantially little transverse width or castingrays with substantial transverse width (e.g., cones or frustums).

The user interface may be projected by the display system as describedherein (such as the display 220 in FIG. 2A). It may also be displayedusing a variety of other techniques such as one or more projectors. Theprojectors may project images onto a physical object such as a canvas ora globe. Interactions with user interface may be tracked using one ormore cameras external to the system or part of the system (such as,e.g., using the inward-facing imaging system 462 or the outward-facingimaging system 464).

FIG. 11 is a process flow diagram of an example of a method 1100 forinteracting with a virtual user interface. The method 1100 may beperformed by the wearable system described herein. Embodiments of themethod 1100 can be used by the wearable system to detect persons ordocuments in the FOV of the wearable system.

At block 1110, the wearable system may identify a particular UI. Thetype of UI may be predetermined by the user. The wearable system mayidentify that a particular UI needs to be populated based on a userinput (e.g., gesture, visual data, audio data, sensory data, directcommand, etc.). The UI can be specific to a security scenario where thewearer of the system is observing users who present documents to thewearer (e.g., at a travel checkpoint). At block 1120, the wearablesystem may generate data for the virtual UI. For example, dataassociated with the confines, general structure, shape of the UI etc.,may be generated. In addition, the wearable system may determine mapcoordinates of the user's physical location so that the wearable systemcan display the UI in relation to the user's physical location. Forexample, if the UI is body centric, the wearable system may determinethe coordinates of the user's physical stance, head pose, or eye posesuch that a ring UI can be displayed around the user or a planar UI canbe displayed on a wall or in front of the user. In the security contextdescribed herein, the UI may be displayed as if the UI were surroundingthe traveler who is presenting documents to the wearer of the system, sothat the wearer can readily view the UI while looking at the travelerand the traveler's documents. If the UI is hand centric, the mapcoordinates of the user's hands may be determined. These map points maybe derived through data received through the FOV cameras, sensory input,or any other type of collected data.

At block 1130, the wearable system may send the data to the display fromthe cloud or the data may be sent from a local database to the displaycomponents. At block 1140, the UI is displayed to the user based on thesent data. For example, a light field display can project the virtual UIinto one or both of the user's eyes. Once the virtual UI has beencreated, the wearable system may simply wait for a command from the userto generate more virtual content on the virtual UI at block 1150. Forexample, the UI may be a body centric ring around the user's body or thebody of a person in the user's environment (e.g., a traveler). Thewearable system may then wait for the command (a gesture, a head or eyemovement, voice command, input from a user input device, etc.), and ifit is recognized (block 1160), virtual content associated with thecommand may be displayed to the user (block 1170).

Additional examples of wearable systems, UIs, and user experiences (UX)are described in U.S. Patent Publication No. 2015/0016777, which isincorporated by reference herein in its entirety.

Example Communications Among Multiple Wearable Systems

FIG. 12 schematically illustrates an overall system view depictingmultiple user devices interacting with each other. The computingenvironment 1200 includes user devices 1230 a, 1230 b, 1230 c. The userdevices 1230 a, 1230 b, and 1230 c can communicate with each otherthrough a network 1290. The user devices 1230 a-1230 c can each includea network interface to communicate via the network 1290 with a remotecomputing system 1220 (which may also include a network interface 1271).The network 1290 may be a LAN, WAN, peer-to-peer network, radio,Bluetooth, or any other network. The computing environment 1200 can alsoinclude one or more remote computing systems 1220. The remote computingsystem 1220 may include server computer systems that are clustered andlocated at different geographic locations. The user devices 1230 a, 1230b, and 1230 c may communicate with the remote computing system 1220 viathe network 1290.

The remote computing system 1220 may include a remote data repository1280 which can maintain information about a specific user's physical orvirtual worlds. Data storage 1280 can contain information useful tosensory eyewear such as a sign language dictionary, auxiliaryinformation source, etc. The remote data repository may be an embodimentof the remote data repository 280 shown in FIG. 2A. The remote computingsystem 1220 may also include a remote processing module 1270. The remoteprocessing module 1270 may be an embodiment of the remote processingmodule 270 shown in FIG. 2A. In some implementations, the remotecomputing system 1220 may be a third party system which is unaffiliatedwith the wearable system 200.

The remote processing module 1270 may include one or more processorswhich can communicate with the user devices (1230 a, 1230 b, 1230 c) andthe remote data repository 1280. The processors can process informationobtained from user devices and other sources. In some implementations,at least a portion of the processing or storage can be provided by thelocal processing and data module 260 (as shown in FIG. 2A). The remotecomputing system 1220 may enable a given user to share information aboutthe specific user's own physical or virtual worlds with another user.

The user device may be a wearable device (such as an HMD or an ARD), acomputer, a mobile device, or any other devices alone or in combination.For example, the user devices 1230 b and 1230 c may be an embodiment ofthe wearable system 200 shown in FIG. 2A (or the wearable system 400shown in FIG. 4) which can be configured to present AR/VR/MR content.

One or more of the user devices can be used with the user input device466 shown in FIG. 4. A user device can obtain information about the userand the user's environment (e.g., using the outward-facing imagingsystem 464 shown in FIG. 4). The user device or remote computing system1220 can construct, update, and build a collection of images, points andother information using the information obtained from the user devices.For example, the user device may process raw information acquired andsend the processed information to the remote computing system 1220 forfurther processing. The user device may also send the raw information tothe remote computing system 1220 for processing. The user device mayreceive the processed information from the remote computing system 1220and provide final processing before projecting to the user. The userdevice may also process the information obtained and pass the processedinformation to other user devices. The user device may communicate withthe remote data repository 1280 while processing acquired information.Multiple user devices or multiple server computer systems mayparticipate in the construction or processing of acquired images.

The information on the physical worlds may be developed over time andmay be based on the information collected by different user devices.Models of virtual worlds may also be developed over time and be based onthe inputs of different users. Such information and models can sometimesbe referred to herein as a world map or a world model. As described withreference to FIGS. 7 and 9, information acquired by the user devices maybe used to construct a world map 1210. The world map 1210 may include atleast a portion of the map 920 described in FIG. 9. Various objectrecognizers (e.g. 708 a, 708 b, 708 c . . . 708 n) may be used torecognize objects and tag images, as well as to attach semanticinformation to the objects. These object recognizers are also describedin FIG. 7.

The remote data repository 1280 can be used to store data and tofacilitate the construction of the world map 1210. The user device canconstantly update information about the user's environment and receiveinformation about the world map 1210. The world map 1210 may be createdby the user or by someone else. As discussed herein, user devices (e.g.1230 a, 1230 b, 1230 c) and remote computing system 1220, alone or incombination, may construct or update the world map 1210. For example, auser device may be in communication with the remote processing module1270 and the remote data repository 1280. The user device may acquire orprocess information about the user and the user's environment. Theremote processing module 1270 may be in communication with the remotedata repository 1280 and user devices (e.g. 1230 a, 1230 b, 1230 c) toprocess information about the user and the user's environment. Theremote computing system 1220 can modify the information acquired by theuser devices (e.g. 1230 a, 1230 b, 1230 c), such as, e.g. selectivelycropping a user's image, modifying the user's background, adding virtualobjects to the user's environment, annotating a user's speech withauxiliary information, etc. The remote computing system 1220 can sendthe processed information to the same or different user devices.

Various functionalities of embodiments of the sensory eyewear system arefurther described below.

Example Sensory Eyewear for Facilitating User Interactions

The wearable system 200 can implement a sensory eyewear system 970 forfacilitating user's interactions with the other people or with theenvironment. As one example of interacting with other people, thewearable system 200 can interpret sign language by, for example,detecting gestures that may constitute sign language, translating thesign language to another language (e.g., another sign language or aspoken language), and presenting the translated information to a user ofa wearable device. As another example, the sensory eyewear system 970can translate speech into sign language and present the sign language tothe user.

The wearable system 970 can also facilitate user's interaction with theenvironment by recognizing objects in the environment, modifying thecharacteristics of the objects (in a virtual environment), andpresenting modified objects as virtual objects to the user. For example,the wearable system 200 can recognize signs (e.g., traffic signs, signsfor store front, etc.) in a user's environment based on images acquiredby the outward-facing imaging system 464, modify the characteristics ofthe sign in the user's environment, and present the modified sign to theuser. The modified sign may be overlaid onto the user's 3D environmentsuch that the original sign maybe occluded.

Example Sensory Eyewear System as a Tool for InterpersonalCommunications

In some situations, one or more people in a conversation may use hand orbody gestures (such as, e.g., a sign language) to express themselves.The conversation may occur during a telepresence session or when thepeople are in the physical vicinity of each other. The wearable system200 can interpret sign language of a signer for a user of the wearablesystem 200 (also referred to as an observer) when the user iscommunicating with the signer. The wearable system 200 can alsotranslate a verbal or sign language based speech into a graphics (suchas, e.g., images of hand gestures) and present the graphics to a signersuch that the signer can understand the speech of the observer. Forexample, an observer wearing a head-mounted display may have a reducedfield of view, and thus the observer may not be able to observe thecomplete gestures made by a signer using a sign language. The wearablesystem 200 can capture the gestures by the signer using theoutward-facing imaging system 464 (because it may have a camera that hasa wider field of view that what a user can perceive through thehead-mounted display). The wearable system 200 can show the capturedgestures as virtual graphics to the observer or show text converted fromthe captured gestures to facilitate the observer's understanding of thesigner's speech. Further, the wearable system 200 can be configured totranslate one sign language into another sign language. For example, oneperson in the conversation may use American Sign Language and the otherperson may use Dogon Sign Language. The wearable system 200 cantranslate the American Sign Language to the Dogon Sign Language for theperson using the Dogon Sign Language and translate the Dogon SignLanguage to American Sign Language for the person who uses the AmericanSign Language.

Example Sign Language Capturing

The wearable system can use various techniques to capture an originalspeech and translate the original speech to a target speech. The speechmay be in the form of hand or body gestures, or audible sounds. Asdescribed herein, the original speech may be in a sign language and thetarget speech may be another sign language or a spoken language.Alternatively, the original speech may be a spoken language while thetarget speech is a sign language. The wearable system 200 can capturethe original speech using the outward-facing imaging system 464, theaudio sensor 232, or by communicating with another computing device viathe network 1290 depending on the context of the speech (e.g., whetherthe speech is in-person or via telecommunications).

As an example of capturing the original speech during an in-personcommunication, where the signer of the detected sign language is in thephysical vicinity of a sensory eyewear system, the outward-facingimaging system 464 can capture images of the user's environment. Thewearable system 200 can detect, from the image information, gestures(e.g., hand/body gestures or lip movements) which may constitute a signlanguage. The wearable system 200 can recognize gestures usingalgorithms such as, e.g., a deep neural network, a hidden Markov model,dynamic programming matching, etc. to recognize signs represented by thegestures made by the speaker. As described with reference to FIG. 7, thegesture recognition may be performed by one or more object recognizers708.

As an example of capturing the original speech in the context of remotecommunications, the wearable system 200 can capture and recognize thepresence of the original speech by analyzing data received from a remotecomputing device (e.g., another wearable device) or by analyzing datacaptured by the outward-facing imaging system 464 (e.g., where theoriginal speech is present on a television). In one example, the signerand the observer may be conversing through an Internet video chatsession. The signer and the observer can each wear their respectiveHMDs. The HMDs can communicate with each other via the network 1290(shown in FIG. 12). Where the signer is in front of a reflected surface(e.g., a mirror), the HMD of the signer can capture the gestures of thesigner by acquiring reflected images of the signer via theoutward-facing imaging system 464. The reflected images of the signermay be sent to the HMD of the observer or the remote computing system1220 for recognition and interpretation of sign languages. As anotherexample, the signer may be a person in a video program, such as onepresented on television or Internet programming, etc. Where the signer'sgestures can be visually observed at the observer's location, a wearablesystem 464 can capture sign language gestures in the same way as it doesin the in-person communication context (e.g., via the audio sensor 232or the outward-facing imaging system 464).

In addition to or in alternative to displaying a text or graphicaltranslation of the sign language gestured by another person to a user ofthe wearable system 200, the user of the wearable system 200 may alsocommunicate with sign language. In this case, the wearable system cancapture the user's own sign language gestures (from a first-person pointof view) by the outward-facing imaging system 464. The wearable systemcan convert the sign language to a target speech which may be expressedin the format of text, audio, images, etc. The wearable system 200 cantransmit the result to another wearable system for presentation toanother user. As described herein, the conversion from the originalspeech to the target speech can be performed by the wearable system ofthe user, another user's wearable system, or the remote computing system1220, alone or in combination. For example, the user's wearable systemcan capture the user's hand gestures and transmit the captured video orimage (containing sign language gestures) to another user's wearablesystem or the remote computing system 120 which can extract signlanguage from the video or image and convert the sign language toaudio-visual content for a speaking language or another sign language.The audio-visual content can include text, graphics, video, animations,sound, etc.

Signer Gesture Rejection and Source Localization

The wearable system can identify a source of gesture or sign languageusing various sensors, such as, e.g., the audio sensor 232, theoutward-facing imaging system 464, stationary input 704, or othersensors in the user's environment. As one example, the wearable systemmay detect a series of hand gestures as well as lip movements from dataacquired by the outward-facing imaging system 464. The wearable systemmay find that the hand gestures are associated with a signer because thesigner also has corresponding lip movements. As another example, thewearable system can measure the distance between the user and thegestures to determine the source of gestures. For example, the wearablesystem can determine that a series of gestures comes from the userbecause the hands appear relatively big in the images acquired by theoutward-facing imaging system 464. But if the hands appear to berelatively small, the wearable system may find the gestures are from aperson other than the user. As yet another example, the wearable systemmay find that the gestures come from an audio-visual content (e.g., in atelevision) by recognizing the object that is playing the audio-visualcontent (e.g., by recognizing the television using the objectrecognizers 708).

Based on the source of the gestures, the wearable system 200 can beconfigured not to process the gestures from certain people. For example,the wearable system may capture gestures from multiple people in theuser's environment but the wearable system can be configured to notprocess the sign language from a person outside the center of the user'sFOV for sign language recognition. As another example, the wearablesystem may be configured not to process the user's own sign language.

In some embodiments, the wearable system can configure the sensors todetect the user's own sign language, such as, e.g., by positioning thecameras in the outward-facing imaging system 464 at an angle such thatthe user does not have to raise his hand in order for the outward-facingimaging system 464 to capture the user's hand gestures. The sensors canalso be configured not to detect the user's own sign language. Forexample, the non-detection can be achieved through not capturing imagesin the direction of the user's own hands (which is typically below theuser's FOV), or filtering out (e.g., by cropping) images in such adirection. Thus the system can distinguish the user's own sign languagefrom those of others.

Example Conversion from Sign Language to Text

The wearable system 200 can convert the captured sign language to textwhich can be presented to a user or translated into another language.Conversion of sign language to text can be performed using algorithmssuch as deep learning (which may utilize a deep neural network), hiddenMarkov model, dynamic programming matching, etc. For example, the deeplearning method (a convolutional neural network in some cases) can betrained on images or videos containing known signs (supervised learning)so as to determine features representative of the signs and to build aclassification model based on the learned features. Such a trained deeplearning method can then be applied by the local processing and datamodule 260 or the remote processing module and data repository 270, 280of the wearable system 200 to images of a signer detected by theoutward-facing imaging subsystem.

The text conversion functionality can be implemented by the localprocessing & data module 260, the remote processing module 270, theremote data repository 280, or the remote computing system 1220, aloneor in combination. For example, the wearable system 200 can includesign-language-to-text functionality implemented on the HMD. As oneexample, the wearable system can store a sign language dictionary in thelocal data module 260 or the remote data repository 280. The wearablesystem can accordingly access the sign language dictionary to translatea detected gesture into text. As another example, the wearable system200 can access sign-language-to-text functionality implemented by theremote computing system 1220. The wearable system 200 may utilizewireless connections to commercial sign-language-to-text services ordata repositories (e.g., via an application programming interface(API)). For example, the wearable system 200 can provide capturedgesture to the remote computing system 1220 and receive thecorresponding text from the remote computing system 1220.

Whether the conversion is performed locally or remotely, otherprocessing steps such as displaying the converted text and retrievingauxiliary information (which are further described below) may be donelocally or remotely independent of where text conversion is performed.For example, if sign-language-to-text conversion is done remotely andthe converted text is to be displayed locally (e.g., the user of thesystem is the observer), a captured video stream can be sent to a remoteprocessing module 270 or a remote server performing the conversion via anetwork; converted text strings are returned to a local component of thesystem (e.g., local processing and data module 260) for display. Asanother example, if sign-language-to-text conversion and auxiliaryinformation retrieval are done remotely, a captured video stream can besent to a remote processing module 270 or to a remote server via anetwork and retrieved auxiliary information can be returned to a localcomponent of the system. Other combinations of local/remote processingare also viable.

Although these examples are described with reference to converting asign into text, the signs may be converted into various other formatssuch as, e.g., graphics, animations, audio, or other types ofaudio-visual content. Further, the translation of the signs does notrequire the signs to be first translated into text.

Examples of Converting One Sign Language to Another Sign Language

As noted herein, there are hundreds of sign languages throughout theworld. Accordingly, the wearable systems described herein can be alsoused when both conversation partners are signing, but in different signlanguage systems. Advantageously, each such signer can use his or herown wearable system to translate the signs of the other signer into theuser's own sign language system. The wearable system may translate thesigns into text understood by the user or into a graphic representationof the user's own sign language.

The wearable system 200 may be configured to recognize a particular signlanguage, e.g., American Sign Language (ASL). The wearable system 200may also be configured to recognize a plurality of sign languages, e.g.,ASL, British Sign Language, Chinese Sign Language, Dogon Sign Language,etc. In some implementations, the wearable system 200 supportsreconfiguration of sign language recognition, e.g., based on locationinformation of the sensory eyewear system. The wearable system mayrecognize a foreign sign language through a means similar to how thesystem recognizes the user's own or preferred sign language, e.g.,utilizing object recognizers 708 alone or in combination with a signlanguage dictionary to recognize gestures perceived by theoutward-facing imaging system 464. The wearable system can convert asign language the user perceives into the user's dominant sign language.The user's dominant sign language can be the user's first sign languageor the user's preferred sign language in a conversation. A sign languageother than the user's dominant sign language can be considered a foreignsign language. The wearable system can allow the user to selectconverted text of the foreign sign language. For example, a user canselect a foreign sign language and the wearable system can present themeaning of gestures in the foreign sign language as text to the user ofthe wearable system.

The wearable system may recognize a foreign sign language through theaid of spoken language in the environment or location information. Forexample, the wearable system detects Italian is spoken in the user'senvironment or determine that user is in Italy based on data acquired bythe GPS. Based on this information, the wearable system canautomatically activate the functions for recognizing the Italian SignLanguage. As another example, the wearable system may have an order ofpreference for sign languages that the wearable system is able tosupport. In this example, ASL may takes precedence over Italian signlanguage because the user is from the United States. However, once thewearable system detects that the user is surrounded by Italian speakersor is physically in Italy, the wearable system can change the order ofpreference so that the Italian sign language is now before ASL. Thus,the wearable system can translate the Italian sign language to Englishtext or graphics associated with ASL.

A sensory eyewear system can not only help a user understand a foreignsign language, it can also help a user sign the foreign sign language.For example, a wearable system can be configured to translate a user'sown language into a foreign sign language. The system can display theforeign sign language gestures (e.g., the translated sign language) onthe display. The user can see the gestures in the foreign sign languageand imitate the gestures. For example, a user may be conversing with asigner who is hearing-impaired. The wearable system can capture theuser's speech and display, to the user, the corresponding gestures in asign language that signer understands. The user can accordingly make thegestures as presented by the display to communicate with the signer. Insome embodiments, rather than showing the gestures to the user, thewearable system can communicate the signs corresponding to the user'sspeech to the signer instead such that the signer is able to understandthe user's vocal speech.

A wearable system can include an audio amplifier (e.g., the speaker 240)to provide the recognized sign language in audio. For example, thewearable system can convert a sign language by a signer into an audiostream for playback to the user of the wearable system 200.

Examples of Determining Auxiliary Information Associated with the SignLanguage

It is not uncommon for people to not know or understand words or phrasesin a conversation, including a conversation involving a sign language.The wearable system can display auxiliary information associated withpart of the displayed text to enhance a user's understanding. Theauxiliary information can include information, such as definition,translation, explanation, etc., which augments and adds to the contextsof the definition. Auxiliary information may present in various forms,such as, e.g., text, image, graphics, animations, or other audio orvisual information. The system can present auxiliary informationvisually, e.g., via the display 220 in FIG. 2A. The system can presentauxiliary information in audio, e.g., via an audio amplifier 240 in FIG.2A, to a user who is not hearing-challenged. By providing a definition,translation, explanation, or other information for such words orphrases, the wearable system advantageously can assist the user inbetter understanding sign language that the user observes.

The auxiliary information may be determined based on contextualinformation of the user's environment, the context of the speech, etc.As an example, the wearable system can utilize, at least in part, userbehavior in determining whether to display auxiliary informationassociated with a conversation partner's signs. For example, a user maytemporarily stare in a certain direction (e.g., toward the signer or thesigner's hands). The wearable system can detect the user's direction ofgaze (e.g., using the inward-facing imaging system 462), and, inresponse, can retrieve and display auxiliary information associated withthe conversation partner's signs.

The wearable system may comprise a data repository (e.g., a database) ofauxiliary information. A wearable system can retrieve auxiliaryinformation associated with displayed text by accessing a datarepository. Such a database of information may be stored locally to awearable device, e.g., in the data module 260 in FIG. 2A, or storedremotely, e.g., in the remote data repository 270. The wearable systemcan utilize publicly accessible information, e.g., information on theInternet, to determine auxiliary information. For example, the wearablesystem can access a network to send a query regarding a word/phrase in aconversation to a resource on the Internet, such as a dictionary, anencyclopedia, or other similar resource. Such resources may be general(e.g., a general purpose encyclopedia such as Wikipedia) or specialized(e.g., an index of drugs such as one on rxlist.com or a mineralogydatabase (e.g., webmineral.com)).

Example Display, Dismissal, and Recall of Converted Signs or AuxiliaryInformation

The wearable system can present the converted signs (e.g., in a text orgraphical format) alone or in combination with the auxiliary informationto a user of the wearable system 200. For example, the wearable system200 can be configured to display auxiliary information together withconverted signs of the sign language, to display converted signs orauxiliary information individually (e.g., displaying only auxiliaryinformation for the duration when auxiliary information is displayed),or to switch between the two display modes. The converted signs alone orin combination with the auxiliary information may sometimes be referredto as the displayed item.

The converted text or auxiliary information may be presented in avariety of ways. In one example, the wearable system 200 can placeconverted text or auxiliary information in text bubbles, e.g., textlocalized geometrically near the signer, such as illustrated in graphic1355 in FIG. 13. As another example, the wearable system 200 can beconfigured to display a rolling transcript of detected sign language. Inthis configuration, words or even sentences that were missed can bequickly reread in case, for example, a user is momentarily distracted. Aconverted text transcript of a signer may be displayed as a rolling textsimilar to presentation of end credit in a movie.

A system displaying a transcript of converted text can highlight a wordor phrase for which auxiliary information is requested in some way,e.g., underlined, colorized, in bold text, etc. Such highlights can bedisplayed before the auxiliary information is retrieved or displayed.Some embodiments configured in this display mode can permit the user toconfirm or cancel the request of the highlighted text. Alternatively oradditionally, such highlights can be displayed together with theauxiliary information. This display mode can make clear to the user thetext to which the auxiliary information is associated. The system canpermit a user to select, through a UI interaction, current or pastconverted text and bring up, or bring back, associated auxiliaryinformation as further described below.

The wearable system 200 may place converted text or auxiliaryinformation (e.g., in text bubbles or as a rolling transcript) so as tominimize a user's eye movement in order to access the information via aUI interaction. In this way, the UI is simplified and the user does notneed to take his or her attention far from the signer. The convertedtext or auxiliary information may be so placed as to make the readingaction minimally visible to a conversation partner and, in so doing,provide less distraction and better communication while not revealingthe user's access to converted text or auxiliary information. Forexample, an implementation capable of determining the location of asigner may place converted text or auxiliary information next to thesigner. Images from the systems outward-facing imaging system 464 canhelp with determination of appropriate placement, for example, notobscuring, e.g., the face, the gesture, etc., of the signer. Thewearable system 200 can use the process flow illustrated in FIG. 8 todetermine the placement of converted text or auxiliary informationdisplay. For example, the recognized object in block 850 can be thesigner whose sign language is to be processed for sign languagerecognition.

As another example of reducing distraction experienced by the user or aconversation partner, if converted text or auxiliary information ispresented in audio (e.g., where the user is not hearing-challenged andthe conversation partner uses sign language), the wearable system canpresent the information at a volume loud enough for the user, but notfor the (speech-challenged but not hearing-challenged) conversationpartner, to hear, or present information when neither the user nor theconversation partner is speaking.

The displayed item may be left visible until a condition is met. Forexample, the displayed item may be left visible for a fixed amount oftime, until the next displayed item is to be displayed, or untildismissed by a user action. The user action may be passive (e.g., eyemovements as captured by the inward-facing imaging system 462). Thewearable system can dismiss a displayed when it determines that the userhas reviewed the displayed item. For example, if the displayed item is atext, the system can track the user's eye movements through the text(e.g., left to right or top to bottom). Once the wearable system hasdetermined that the user has looked through the entire displayed item(or a majority of the displayed item), the wearable system canaccordingly dismiss the displayed item. As another example, a displayeditem may be dismissed after the user has been observed by the system tohave looked away from (or to not look at) the area occupied by thedisplayed item. The user action may also be active (e.g., by a handgesture as captured by the outward-facing imaging system 464, a voiceinput as received by the audio sensor 232, or a input from the userinput device 466). For example, once the wearable system detects a swipegesture by the user, the wearable system can automatically dismiss thedisplayed item.

A wearable system can be configured to support a customized set of userinterface (UI) interactions for a particular user. UI interactions maytake the form of a UI element analogous to a button that is actuatedeither with a finger, a pointer or stylus of some kind, by the gaze andsubsequent fixation on the button with the eyes, or others. The buttoncan be a real physical button (e.g., on a keyboard) or a virtual buttondisplayed by the display 220. UI interactions may take the form of ahead pose, e.g., as described above in connection with FIG. 4. Anexample of UI interaction detection is described above in connectionwith FIG. 10.

A wearable system can prompt a user to delay dismissal of a displayeditem. For example, the wearable system 200 may reduce the brightness orchange the color scheme of the displayed item to notify the user thatthe displayed item will be dismissed shortly, e.g., a few seconds. A UIinteraction such as those described above may be used to postpone thedismissal. For example, a wearable system may detect that a user haslooked away from a displayed item. Thus, the wearable system canincrease the transparency of the displayed item to notify the user thatthe displayed item will be dismissed shortly. However, if the wearablesystem, through eye tracking, detects that the user looks back to thedisplayed item, the AR system can postpone the dismissal.

A UI interaction such as those described above may also be used torecall a displayed item that has been dismissed. For example, an inputaction through the user input device (e.g., an actuation of a backspaceon a keyboard) can be used to recall the most recent displayed item, orused to select a particular displayed item for recall.

Example User Experiences of a Sensory Eyewear System

FIG. 13A shows an example user experience of a sensory eyewear systemwhere the sensory eyewear system can interpret a sign language (e.g.,gestured by a signer) for a user of a wearable system. This exampleshows a signer 1301 who the user of a sensory eyewear system isobserving. The user can perceive that the signer 1301 is making asequence 1300 of hand gestures as shown in the scenes 1305, 1310, and1315. The hand gesture in the scene 1305 represents the word “how”; thehand gesture in the scene represents the word “are”; and the handgesture in the scene 1315 represents the word “you”. Thus the sequence1300 can be interpreted as “How are you”. The sequences 1320 and 1340show the same gestures as the sequence 1300. The gesture 1305corresponds to the gestures 1325 and 1345; the gesture 1310 correspondsto the gestures 1330 and 1350; and the gesture 1315 corresponds to thegestures 1335 and 1355. However, the sequences 1300, 1320, and 1340illustrate different user display experience as further described below.

To translate the hand gestures in the sequence 1300 to the Englishphrase “How are you”, an outward-facing imaging system 464 of thewearable system 200 can capture the sequence of gestures, either as aseries of images or as a video. The wearable system can extract gesturesfrom the series of images or the video. The wearable system can performsign language recognition on the extracted gestures, for example,through object recognizers 708 or applying a deep learning algorithm. Inthe process or recognizing sign language, the wearable system can accessa sign language dictionary stored in a local or a remote storage. Thewearable system can display text (or a graphic representation of thesign) converted from the recognized sign language to the user (notshown) via the display 220. The sensory eyewear system can also receivea request for auxiliary information associated with the converted signs,and retrieve and display the auxiliary information use the techniquesdescribed herein.

In the graphical sequences illustrated in FIG. 13A, expressing the word“how” takes two distinct gestures, e.g., as shown in graphics 1305 and1310. The wearable system may wait until after the second gesture (inthe scene 1310) before displaying the word “how” (as gestured in thescene 1305). Additionally or alternatively, the wearable system may holdoff on text conversion or display until a sentence or phrase iscompleted, such as shown in graphical sequence 1320, where the phrase“How are you” is shown at the end of the scene 1335. The wearable systemcan display converted text or auxiliary information as a caption or as atext bubble, e.g., as shown in graph 1355. The caption or the textbubble can be positioned in the user's FOV to minimize distraction tothe user, e.g., in close proximity to the signer without of obscuringthe user's view of the signer's face.

FIG. 13B shows another example user experience of a sensory eyewearsystem, where the target speech and auxiliary information are bothpresented. In this example, a user (not shown) can wear an HMD andperceive a signer 1362. The signer asks the question “Where is the PTO?”using a sign language (the signer is depicted as gesturing the letter“O” at the end of the question). The wearable system can recognize thegestures made by the signer, convert them to text, and display theconverted text in a text bubble 1360 to the user of the wearable system.The wearable system can determine that the “PTO” is an acronym and is aword that the user does not use often in everyday speech. For example,the wearable system can maintain a dictionary of commonly used words andphrases and determine that “PTO” is not in the dictionary. Upondetecting that the word “PTO” is not in the dictionary, the wearablesystem can initiate an access of auxiliary information associated withthe phrase “PTO”.

The wearable system can retrieve auxiliary information on the acronymbased on contextual information. In this example, the system may rely onits location information, e.g., the system (and its user) is presentlyin Alexandria, Va. The system retrieves “Patent and Trademark Office” asauxiliary information for the acronym “PTO.” The system displays theauxiliary information as a virtual banner 1365 to the user via thedisplay 220. The display mode of the converted text and the auxiliaryinformation shown in FIG. 13B are for illustration only. Someembodiments may display them differently, e.g., both in a captiondisplayed sequentially.

A plurality of users of a plurality of wearable systems can communicateremotely through the aid of their respective wearable systems. FIG. 13Cshows an example user experience of a sensory eyewear system in atelepresence session. For example, as illustrated in FIG. 13C, two users1372 a, 1372 b at two physical locations 1370 a, 1370 b (such that theydo not see or hear each other directly, without the aid of a man-madedevice) can both wear a wearable device 1374 a, 1374 b respectively. Oneor both users 1372 a, 1372 b may converse using a sign language. Thehand gestures may be captured by an imaging system of the users'respective wearable system and transmitted through the network 1290.User A's 1372 a sign language may be displayed as converted text on userB's 1372 b device, and vice versa.

A sensory eyewear system can convert detected sign language to textlocally and transmitted only the converted text through the network1290. The other user's device can either display the text or, where theother user is not hearing-challenged, convert the text to audiblespeech. This can be advantageous where the bandwidth of the network 1290is constrained because a smaller amount of data is required to transmittext than to transmit corresponding images, video, or audio.

A wearable system can also enhance a telepresence conversation throughimages presented on the display 220. For example, the display 220 canpresent an avatar of a remote signer along with converted text orauxiliary information to engage a participant's visual senses. Forexample, a wearable device equipped with an inward-facing imaging system464 can capture images for substituting the region of a wearer's faceoccluded by an HMD, which can be used such that a first user can see asecond user's unoccluded face during a telepresence session, and viceversa. World map information associated with a first user may becommunicated to a second user of a telepresence session involvingsensory eyewear systems. This can enhance user experience through thecreation of images of the remote user to be seen by an HMD wearer.

In a telepresence application, capturing image information is performedby a device associated with a user-signer (e.g., from a first personpoint of view), rather than a device associated with a user-observer,which may be typical in the in-person scenario. Detection of presence ofsign language and conversion of sign language to text can be performedby device associated with either user, or by a remote system, e.g.,server computer system 1220. The source of sign language can bedetermined based on the device that captures the image, e.g., when userA's device captures the image, user A is signing.

FIG. 13D illustrates an example virtual user interface for interpretinga sign language. In this example, the user 1392 is wearing a wearabledevice 1380 (which may include at least a portion of the wearable system200). In this example, the user 1392 is behind a counter and perceives aperson 1394 approaching the counter. For example, the user 1392 may be anurse or admittance person in a medical facility, a hotel employee(e.g., concierge) who assists guests, and so forth. The person 1394 maybe feeling unwell and seeking medical attention such as directions to apharmacy. The wearable device 1380 can observe (e.g., via theoutward-facing imaging system 464) the hand gestures by the user 1394 asshown in FIG. 13D. The wearable device 1380 can automatically (e.g.,using object recognizers 708) detect that the hand gestures as shown arean expression in a sign language, recognize the meaning associated withthe hand gestures, and provide the translation of the hand gestures in atarget language (e.g., English) which the user 1392 understands. Thewearable device 1380 can present a virtual user interface 1382 to showthe input 1384 a captured by the wearable device, the translation 1384 bcorresponding to the input 1384 a (e.g., “Is there a pharmacy nearby?I'm feeling unwell.”). The wearable system can also provide user inputelements 1384 c and 1384 d on the virtual user interface 1382. Forexample, the user 1392 may use a hand gesture (e.g., a press gesture) toselect the user input element 1384 c. An actuation of the user inputelement 1384 c may cause the wearable device to provide a list ofresponses, such as, e.g., the location of the nearby pharmacy, or “Idon't know”. In some embodiments, the wearable device 1380 can showcorresponding graphics in the sign language for the responses. The user1392 can accordingly respond to the person 1394 using hand gestures asshown in the graphics. As another example, where the user input element1384 d is actuated, the wearable system can provide list of options suchas, e.g., dismissing the user interface element 1382, or making a callfor help, etc. In some embodiments, the area 1384 a of the interface1382 may include an output graphic, showing sign language gestures thatthe user 1392 can perform to communicate with the person 1394 (e.g.,signs for “the pharmacy is across the street”).

Example Processes for a Sensory Eyewear System as a Tool forInterpersonal Communications

FIGS. 14A and 14B illustrate example processes for facilitatinginterpersonal communications with a sensory eyewear system. The exampleprocesses 1400 and 1440 in FIGS. 14A and 14B can be performed by thewearable system shown in FIG. 2A.

At block 1404, the wearable system can capture image information in anenvironment. As described herein, the wearable system can use theoutward-facing imaging system 464 to capture image information in theuser's surroundings. The wearable system can also capture the audioinformation in the environment. The audio information can be used withthe data acquired by the outward-facing imaging system 464 to determinea source of the speech or gesture, or detect the presence of a signlanguage.

At block 1408, the wearable system detects the presence of sign languagein the captured image information. This detection processing may be donelocally (e.g., by local processing module 71) or remotely (e.g., byremote processing module 72). The wearable system can use various objectrecognizers to detect the presence of hand gestures. For example, thewearable system may find a sequence of hand gestures may constitute aphrase or a sentence in a sign language. As another example, thewearable system may detect a series of hand gestures as well as lipmovements. The wearable system may find that the hand gestures and lipmovements are associated with a sign language because such gestures andlip movements are not accompanied by audio information.

In some embodiments, the wearable system can detect and interpret a signlanguage based on contextual information. For example, the wearablesystem can receive audio signals (e.g., of dinner conversation), convertthose signals into language, or extract meaning from that language,thereby inferring the genre (or other attribute) of the topics ofdiscussion which can be used to interpret the sign language (such as,e.g., to interpret hand gestures in a way to align with the topics ofdiscussions).

The wearable system can be configured to detect or to ignore the user'sown sign language. The function of block 1408 can be different based onthis configuration because the user's own sign language can be capturedfrom a first-person point of view at a relatively close distance. Forexample, if the system is configured to capture the user's own signlanguage, an additional outward-facing camera directed downward at theuser's hands may be turned on, or the outward-facing imaging system maybe configured into a wide-angle mode to capture images of the user'shands.

At block 1412, the system determines whether sign language is detected.If sign language is detected, the process flow 1400 continues to block1416. If sign language is not detected, the flow returns to block 1408(as shown) or to block 1404 (not shown).

The operations in blocks 1404 through 1412 may be performed continuouslyor periodically (e.g., at a sampling frequency) when the wearable system(including its imaging systems) is turned on or when the sign languagerecognition function is enabled. These operations can be performed inparallel to other blocks in flowcharts 1400 and 1440 (e.g., asbackground tasks driven by a timed interrupt). They are shown asdiscrete blocks in a processing flow sequence for the purpose ofillustration. But they are not limited by the illustrated sequence. Manyprocessing flows other than the examples described above are possible atthe discretion of a system designer.

At block 1416, the wearable system can determine a source (e.g., thesigner) of the detected sign language. The source may be a person in thephysical vicinity of the user, the user, or a person in a visual contentthat the user perceives. The source of the sign language can berelevant, for example, if the system is configured to process signlanguage only from a person in or near the center of the wearablesystem's FOV (sign language from people outside of the center of theFOV, e.g., when multiple persons are concurrently conversing in signlanguage, can be discarded and not processed further). As anotherexample, the wearable system can process gestures for sign languagerecognition only for a person at whom the user is looking, which may ormay not be a person at the center of the FOV. The wearable system canidentify the person which the user is looking at based on data acquiredby the inward-facing imaging system 462 and the outward-facing imagingsystem 464. For example, an outward-facing camera can provideinformation including position of a signer relative to the user. Aninward-facing camera can provide information including the direction inwhich the user is looking. By using information from both cameras, thewearable system can determine the person whom a user is looking at, andwhether that person is the source of sign language.

At block 1420, the system determines whether there has been a change inthe source of sign language. If there has been a change, the flow 1400continues through block 1424 to block 1444 as shown in FIG. 14B. Ifthere has not been a change in the source of sign language, the flowmoves to block 1428 to continue sign language recognition processing,which can include capturing image information (block 1404), detectingpresence of sign language (block 1408), as well as processing stepsshown in FIG. 14B. For example, if the system determines the gesturescontinue to come from the same signer continues, the system can continueto perform functions starting from block 1448 in addition to continuingto capture image information and detect sign language.

At block 1448, the wearable system can translate the sign language intoa language understood by the user. For example, the system can convertthe recognized sign language to text, which can be read by the user whendisplayed by the system (e.g., as a text bubble or caption). In somecases, if the user understands a different sign language, a graphicrepresentation of the other signer's signs can be displayed to the user,for example, as graphics showing the signs converted into signs in theuser's own sign language.

At block 1452, the example system can determine whether the detectedsign language is the user's own, when the system is configured to detectthe user's own as well as a conversation partner's sign language. If itis, the process proceeds to block 1484, wherein the system can transmitthe converted text to a display device of the observer/conversationpartner.

From block 1484, the system can proceed to block 1488 to continueprocessing. When the system is configured to ignore the user's own signlanguage, both blocks 1452 and 1484 can be omitted from the flow. If thedetected sign language is not the user's own, the flow continues toblock 1456.

At block 1456, the wearable system can display converted text by thedisplay as described above. Where the user of the system is nothearing-challenged, the text can be presented in audio, e.g., throughthe audio amplifier 240, in addition to or in alternative to the visualdisplay.

At block 1460, the wearable system can monitor for a request forauxiliary information on the converted text. The request for auxiliaryinformation may be sent by the user's wearable device upon detection ofa triggering condition. Some example triggering conditions may include auser's indication, e.g., a user's gesture or an actuation of the userinput device 466; or upon detecting of a word (or phrase) that a usermay not understand.

At block 1464, the system determines whether a request is received. If arequest is not received, the flow moves to block 1476, which is furtherdescribed below.

If a request is received, at block 1468, the system can retrieveauxiliary information associated with the converted text (or a requestedportion thereof). As described herein, the auxiliary information may bedetermined and retrieved based on contextual information, such as, e.g.,the user's location, the context of the speech, or other types ofinformation as described herein.

At block 1472, the wearable system can display retrieved auxiliaryinformation via the display 220 of the wearable system. In someimplementations, the wearable system may dismiss a display of theconverted text before displaying auxiliary information.

The flow may enter block 1476 from block 1464 or 1472. At block 1476,the system can detect a condition for dismissing the converted text orauxiliary information display. When such a condition is detected, atblock 1480, the system can dismiss the display of the converted text orauxiliary information and continue on to block 1488. At block 1488,processing of sign language recognition continues, in a manner similarto the description of block 1428 above.

Similar to what is noted above with respect to blocks 1404 through 1412,operations in flowchart 1440 may be performed in parallel to otherblocks in flowcharts 1400 and 1440. They are shown as discrete blocks ina processing flow sequence for the purpose of illustration, but they arenot limited by the illustrated sequence. For example, a system may bedisplaying auxiliary information for converted text (at block 1472)while the system converts additional sign language to text (at block1448), performs auxiliary information request monitoring (at block1460), or retrieves auxiliary information on another converted text (atblock 1468). As another example, a system can convert sign language totext (at block 1448) while it retrieves auxiliary information for aprior-requested (for auxiliary information) converted text (at block1468). Many other processing flows are possible at the discretion of asystem designer.

FIG. 14C is a process flow diagram of an example method for determiningauxiliary information and presenting the auxiliary informationassociated with converted text. This process 1490 can be executed on thewearable system 200 described herein or another computing device whichitself may or may not have sign language recognition functionality. Thisprocess 1490 can be applicable to situations where it is moreadvantageous to detect sign language and convert the sign language totext using one sensory eyewear system and to display the converted texton another device or system. An example situation can be where a signerwishes to communicate remotely with a second person. The wearable deviceof the signer can convert the signer's own sign language to text. Thewearable device can transmit the converted text to a remote systemviewable by the second person. Since converted text can be transmittedin much fewer information bits than the corresponding images or video,such a process can advantageously require a much lower bandwidth fromthe transmission medium or result in a much more reliable communication.

The process 1490 starts at block 1492, wherein the device or system isperforming some sort of processing, which may or may not be related tosign language processing. At block 1494, the device or system candetermine whether text is received from a wearable system. If no, theprocess can return to block 1492. If yes, the process can proceed toblock 1496. At block 1496, the device or system can receive text fromthe wearable system and render the text. The process can then proceed toblock 1456. Where the rendering device comprises an HMD, the renderingdevice can present the text as virtual content overlaid on the physicalenvironment of the user. Processing in blocks 1456 through 1480 canproceed similarly as described above in connection with FIG. 14B.

FIG. 15 illustrates another example process for facilitatinginterpersonal communications with a sensory eyewear system. The exampleprocess 1500 may be performed by one or more components of the wearablesystem 200 (e.g., by the local processing & data module 260, the remoteprocessing module 270, alone or in combination) described herein. Asdescribed with reference to FIG. 12, one or more of the steps describedin this FIG. 15 can be performed by one or more computing devices thatare not part of a user's wearable system, such as, e.g., another user'swearable device, or a third party's server system.

At block 1510, the wearable system can identify and recognize speech inan environment. The speech may be in the form of a sign language. Forexample, the wearable system can analyze data acquired by theoutward-facing imaging system 464 to identify hand gestures that arepart of a sign language. The wearable system can also analyze audio dataacquired by the audio sensor 232 which may include speech by a person inthe user's environment. The wearable system can recognize the speechusing object recognizers 708. For example, the wearable system canrecognize the presence of a phrase or a word by analyzing images of thesign language using the object recognizers. The wearable system can alsorecognize the audio data using the various speech recognition algorithmsdescribed in FIG. 7.

At block 1520, the wearable system can identify a target language. Thetarget language may be the language a user of the wearable system usesto communicate. For example, the user may communicate with other peopleusing English while the recognized original speech (used by anotheruser) is a sign language. The target language may also be a languageselected by the user or by the wearable system. For example, the usermay select ASL as the target language because the user may want to usesign language to communicate with another person even though the userspeaks another language. As another example, the wearable system mayautomatically select a language based on the user's location. Forexample, the wearable system can determine which country the user is atand select an official language of that country as the target language.

At block 1530, the wearable system can convert the detected speech intothe target language. The wearable system can use various techniquesdescribed herein, such as, e.g., dictionary translations, to performsuch conversion.

At block 1540, the wearable system can determine audio-visual contentassociated with the converted speech for presentation to a user of thewearable system. As one example, the audio-visual content may includetext in the target language. As another example, the audio-visualcontent may be an audio stream in the target language where theconverted speech is in a spoken language. As yet another example, theaudio-visual content may be graphics or animations if the targetlanguage is a sign language.

At the optional block 1550, the wearable system can communicate theaudio-visual content to a head-mounted display for presentation. Forexample, the audio-visual content may be communicated from one user'swearable device to another user's wearable device. In this example, thewearable device of the first user can capture the first user's, convertthe speech to a target language, and communicate the converted speech tothe second user's wearable device.

Example Sensory Eyewear System as a Tool for Interacting withEnvironment

In addition to or as an alternative to recognizing gestures by anotherperson, the wearable system described herein can also recognize signs inthe environment with, for example, various text recognition algorithmsdescribed with reference to FIG. 7. The wearable system can also modifythe text (e.g., modify the display characteristics or the content of thetext) and render the modified text onto the user's physical environment.For example, the modified text may be rendered to overlay and occludethe original text such that the user will perceive the modified textrather than the original the text.

Examples of Modifying Display Characteristics of the Text

FIGS. 16A-16E illustrate example user experiences for a sensory eyewearsystem which is configured to recognize a text in the environment,modify the display characteristics associated with the text, and renderthe modified text. With reference to FIG. 16A, a user 210 can wear awearable device (not shown in FIG. 16A) and may see a physical object1606 in the environment via the display 220. The wearable device caninclude an outward-facing imaging system 464 which can capture an image1602 that comprises the object 1606 within the image 1602. In additionto or in alternative to the outward-facing imaging system 464, thewearable system 200 can capture an image of the physical object usingother sensors or devices. For example, a user input device 466 (e.g., atotem) may have imaging capacities and can capture the image 1602 whichincludes an image of the object 1606. The object 1606 may include a signor other object that may contain writing, letters, symbols, orcharacters 1610 on or in it. For example, the letters may be written onthe object; or shaped from, with, or embedded in the object. The textmay also be a sequence of static or flashing lights; or an arrangementof one or more physical objects. In the examples shown in FIGS. 16A-16E,the object 1606 is a traffic stop sign. In other examples and withoutlimitation, the object 1606 could be any type of signage (e.g., acommercial or public display sign), a book, a magazine, a piece ofpaper, a computer display screen, a television screen, and so forth.

The wearable system 200 can analyze the image 1602 and recognize theobject 1606 using one or more object recognizers 708, for example asdescribed with reference to FIG. 7. As one example, the wearable systemcan recognize that the object 1606 is a traffic sign (e.g., based on theshape of the object 1606, an octagon in FIG. 16A). As another example,the wearable system can recognize the presence of the text in the object1606. The wearable system can recognize the text regardless of theformat of the text (e.g., whether the text is on the object or isrepresented by a sequence of lights that project the text (e.g., neonlights, LED lights, etc.).

As will be further described with reference to FIG. 18, in certainembodiments, the wearable system 200 can recognize the meaning of thetext and convert the text from an original language to a targetlanguage. For example, the wearable system 200 can identify letters,symbols, or characters from a variety of languages, such as, forexample, English, Chinese, Spanish, German, Arabic, Hindi, etc., andtranslate the text from the original, displayed language to anotherlanguage. In some embodiments, such translation can occur automaticallyaccording to previously specified settings (such as, e.g., user'spreference or user's demographic or geographic information). In someembodiments, the translation can be done in response to a command (e.g.,verbal or gesture) from the user.

The wearable system 200 can analyze the characteristics of the text 1610using the object recognizer 708. For example, the wearable system 200can recognize the font size or typeface associated with the text 1610.The wearable system can adjust the characteristics of the text 1610 togenerate a modified text. For example, the wearable system 200 mayadjust the size of the text 1610 to magnify or shrink the text 1610. Thesize of the modified text may be dependent in part on a distance fromthe eye 210 to the original text 1610 or user's characteristics. Forexample, if the text 1610 is far away from the user, the wearable systemcan enlarge the text 1610. As another example, depending on the user'seye capability, the system can make a determination on how to adjust thesize of the text. The wearable system can determine the person's eyecapability based on information acquired previously from the user. Forexample, the user can input whether there are any vision problems of theeyes. The wearable system can also perform a vision test to the user(e.g., by displaying virtual objects at different depth planes and sizesto determine if the user can clearly perceive the virtual objects) todetermine the user; s eye capacity. Based on the user's eye capacity,the wearable system can determine whether the user will likely perceivethe text 1610 based on the characteristics (e.g., the distance/location,color, size, font, etc.) of the text. For example, the wearable systemcan enlarge or bold the text if the wearable system determines that theuser cannot perceive the text clearly (e.g., when the text is out offocus). If the user is near-sighted, but the text is far away from theuser, the wearable system can enlarge the size of the text so that theuser can more easily perceive the text. The size adjustment maycorrespond to the degree of near-sightedness. The size may be associatedwith a larger increase if the degree of near-sightedness of a user isbig while the size may be associated with a smaller increase if thedegree of near-sightedness of the user is small. As further describedherein, the wearable system can also change the display location of themodified text based on the user's eye capacity. With reference to FIG.3, the display system 220 can include a plurality of depth planes, Wherethe user is far-sighted but the text is close to the user, the wearablesystem can render a modified text at a depth plane 306 farther away fromthe user than the original depth plane such that the modified textappears far away from the user. The size adjustment can occur bychanging the front size of the text (e.g., where the text is recognizedas a string). The size adjustment can also occur by zooming in or out(e.g., digital zoom) on a portion of the image 1602 containing the text1610 (e.g., where the text is analyzed as an image rather than a textstring).

The wearable system 200 can render the modified text to a user. Withcontinued reference to FIG. 16A, a user wearing the HMD may see avirtual image 1614 (as rendered by the HMD) containing a renderedversion 1618 of the object 1606. In some implementations, the renderedversion 1618 of the object 1606 can occlude the original text. As shownin FIG. 16A, the rendered text 1622 is “STOP” and is enlarged incomparison to the original text 1610. The HMD can render the enlargedtext 1622 to be overlaid on the original text 1610 and thus the user maynot perceive the original text 1610. In this example, by increasing thetext size, the user advantageously can more readily perceive,understand, and respond to the underlying text 1610, which in actualitymay be much smaller and harder to perceive.

FIG. 16B illustrates another example of modifying characteristics of atext in the user's environment. As shown by the rendered text 1634, thewearable system 200 can bold the font of the original text 1610. Inaddition to or in alternative to bolding the font, other alterations tothe original text 1610 can be made as well, such as, for example,changing the text color, shading, outlines, format (e.g., italics,underline, alignment, justification, etc.), and so forth. The wearablesystem 200 may add (or modify) graphics elements associated with thetext 1610 such as making the rendered text 1634 flash, spin, etc.

FIG. 16C illustrates an example of rendering the modified text togetherwith a focus indicator 1640. The focus indicator 1640 can comprisevisual effects, such as, a bulls eye, a cross-hair, a halo, a color, aperceived depth change (e.g., causing the rendered text to appearcloser), an addition or a change in the background of the text, ananimation, or other visual effects which draw the user's attention. Inthe example shown in FIG. 16C, the wearable system 200 may be configuredto display the focus indicator 1640 as a background 1650 against whichthe text 1638 is rendered. The background 1650 can comprise a borderregion 1642 and an inner region 1646. The border region 1642 can boundthe inner region 1646. In the embodiment shown, the virtual letters 1638are displayed within the inner region 1646. The text background 1650 canbe rendered in the displayed image 1614 such that the text background1650 is a different background than the user would see without the HMD.In some embodiments, one or more of the inner region 1646 and the borderregion 1642 are monochrome (e.g., white, black, or gray). The system canalter the background 1650 such that the rendered text 1638, rather thanthe original text 1610, is seen by the user. For example, the backgroundmay be non-transparent such that it can occlude the original text 1610.The processing electronics can also be configured to display thebackground 1650 such that it blends into the rest of the image 1614. Forexample, the background 1650 may have the same color and texture effectas the rest of the image 1614. The wearable system can also display thebackground 1650 and the text 1638 in a way that highlights the text 1638or background 1650, such as, e.g., a displaying a halo around the text1638 or the background 1650. In such cases the background 1650 may notseamlessly integrate into the rest of the image 1614. For example, theinner region 1646 can be outlined by the border region 1642 in order toemphasize the background 1650 or the text 1638.

Under certain circumstances, the visual appearance of the original textmay not be clear, for example, because of environmental effects (e.g.,rain, fog) between the user and the object 1606. FIG. 16D illustrates anexample of modifying characteristics of the text and rendering themodified the text so as to be more legible. In this figure, the text1626 appears to be blurry to the user 210. Text may be perceived blurryfor a variety of reasons. For example, a user with poor eyesight mayhave trouble seeing text clearly at a particular distance. Users withmyopia may find images of text that are nearby to appear relativelyclear while text that appears far away to be blurry. Similarly, thosewith hyperopia can see text appearing far away clearly while having ahard time with putting into focus text that appears to be nearby. Buteye conditions may not be the only reason an image may appear blurry.Text that appears to be closer or farther away than the eye 210 canaccommodate for may also appear blurry. If the text appears to be movingrapidly relative to the user, the text 1626 may appear to be blurry.Other factors described above, such as climate or weather factors, aswell as resolution of cameras which acquired the images may also play arole.

In this example, the wearable system 200 can make the blurry text 1626or text that is otherwise difficult to read clearer or more legible.Where the text appears to be blurry to the user but not in the imagesreceived by the wearable system, the wearable system can analyze imagesacquired by the outward-facing imaging system 464 or another device(such as e.g., the user input device 466 or a camera external to thewearable system such as a dash cam) to identify the text 1626 usingsimilar techniques described with reference to FIG. 13A. The wearablesystem can render the text virtually as shown by the text 1630. Incertain implementations, the wearable system can adjust thecharacteristics of the virtual text 1630 based on the user's or theenvironment's conditions. For example, where the user is near sighted,the wearable system can enlarge the font of the text 1626 or render thetext to appear closer to the user (e.g., on a closer depth plane). Asanother example, when the environment is dark, the wearable system canincrease the contrast ratio between the text 1630 and the other regionsof the virtual image 1614.

In some situations, the text 1626 appears to be blurry because the image1602 obtained by the wearable system is blurry (e.g., due to fastdriving speed or when the camera's resolution is low). As describedherein, the wearable system can use the object recognizer 708 toidentify the existence of the blurry text 1626. For example, thewearable system can determine a likelihood on the existence of the textin or on the object 1606. In some situations, if the likelihood passes athreshold, the wearable system can use one or more text recognitionalgorithms described, e.g., with reference to FIG. 7, such as an OCRalgorithm, to identify letters 1630 that most likely correspond to theblurry text 1626.

FIG. 16E illustrates a scenario when the original text 1610 is partiallyillegible due to an obstruction 1654. As shown, the obstruction 1654covers part of the original text 1610 in the original image 1602.However, the obstruction 1654 can take on one or more various forms. Forexample, the obstruction 1654 could be some physical obstruction betweenthe eye 210 or display and the image 1602, such as, for example, a pole,a building, etc. The obstruction 1654 could also be an environmental orweather obstruction, such as those described above. The obstruction 1654could also be on the object 1606 (e.g., a portion of the text 1610 isoccluded by another object on the sign 1606 or a portion of the text1610 is erased, missing, or covered by a sticker). This could include,for example, a surface that has accumulated dust or dirt, damage to thesurface of the object 1606 where the writing 1610 is found, an inkblot(e.g., from a printer), a distortion in the original text 1610, or anyother similar obstruction 1654.

The system may use contextual information (also sometimes referred toherein as context clues) in determining what the original text 1610says. The various context clues described herein may be usedindividually or in combination by the wearable system to determine thefull text for the text 1610. An example context clue is the user'slocation. For example, as described above, the GPS system 37 (see FIG.2B) can acquire location data of the user and based on the locationdata, the wearable system can provide an initial guess as to what thelanguage of the text is. Wherein applicable, in some embodiments, thewearable system may gain additional information from signals receivedfrom one or more light sources at wavelength(s) outside the visiblespectrum (e.g., infrared, ultraviolet). For example, the wearable systemmay emit an ultraviolet light toward the sign 1606 to reveal the signageinformation that is only visible under the ultraviolet light (or maydetect ultraviolet light reflected by another source (e.g., the sun)from the signage). In some embodiments, the system has access to adatabase of words against which the system can check the visible portionof the original text 1610. In such examples, the wearable system 200 maybe able to determine which candidates of letters or words are mostlikely. For example, as shown in FIG. 16E, the system infers that theletters spell “STOP” in part due to the octahedral shape of the object1606 or the red color (not shown) of the object 1606.

The wearable system may be able to rely on surrounding words, symbols,punctuation, or characters as context clues to determine what anoriginal text 1610 says. In certain embodiments, the system is able toidentify location-specific context clues using, for example, machinelearning techniques. For example, the system may be able to detect thata user that is driving down the street and may bias an identification oftext towards words frequently used on street signs. The wearable systemmay comprise a database, which may be accessed by the local processingand data module 270 or the remote processing module 280 (see, e.g., FIG.2A). The database may store categories of words that are associated withparticular activities engaged in by the user (e.g., skiing),geographical locations of the user, speeds of travel of the user,altitude of the user, volume or type of ambient noise received by thesystem, level or type of visible or other light in the area received bythe system, the temperature or climate surrounding the system, theperceived distance of the text from the user, or the category orcategories of words spoken by another party that the system picks up. Insome embodiments, the wearable system can use this information ascontext clues to more accurately hone in on more likely candidates forthe words or language of the text viewed by a user according to one ormore associations described above. In some embodiments, the wearablesystem can use machine learning algorithms (e.g., a deep neural network)to “learn” from previous words in various circumstances and identity thelikely word based on the present circumstance. Accordingly, byperforming this learning, the wearable system 200 can becomeparticularized to the behavior of the user and can more rapidly orefficiently determine the text.

In the examples described in FIGS. 16A-16E, the system can determine atwhich depth to display the text based on the perceived distance theoriginal letters appear to be from the user. The perceived distancebetween the original letters and the user may be measured using varioustechniques such as by applying a stereo vision algorithm (e.g., on thedata acquired by the outward-facing imaging system) or by analyzing dataacquired by a depth sensor (e.g., a lidar). Stereo vision algorithms caninclude a block-matching algorithm, a semi-global matching algorithm, asemi-global block-matching algorithm, disparity maps, triangulation,depth maps, a neural network algorithm, a simultaneous location andmapping algorithm (e.g., SLAM or v-SLAM), and so on. Letters that areperceived to be close to the user may be displayed at a near depth onthe display system 220. In some embodiments, letters that appear to becloser than a first distance threshold (e.g., about 800 cm) from theuser are displayed on the system at a first depth. In some embodiments,the first distance threshold is 200 cm, such that letters that appear tobe closer than about 200 cm are displayed at the first depth. In someembodiments, the first distance threshold is about 80 cm. Whetherletters are displayed as if at the first depth or what first distancethreshold is used may depend on a number of factors. One factor may beat how many different depths the system is able to display. For example,a shorter first distance threshold may be used if the embodiment onlydisplays objects at two different depths, whereas a smaller range may beused when the embodiment can display the text at a greater number ofdifferent depths. For example, if a user is reading the newspaper, thesystem will perceive the text to be close to the user, so the letters onthe newspaper will be displayed on the system as if at a close depth. Asshown in FIG. 3, the display system 220 may comprise a plurality ofdepth planes 306 which can cause the virtual objects to appear atdifference distances from the user. In certain implementations, thewearable system can adjust the rendering location of the modified textbased on the user's eye capacity. For example, where the user isnear-sighted, the wearable system can render the modified text at adepth plane closer to the user than the depth plane of which textoriginally corresponds to. As another example, where the user isfar-sighted, the wearable system can render the modified text at a depthplane farther away from the user than where the original text appears.

Similarly, letters that are perceived to be far from the user may bedisplayed at a far depth on the display system. In some embodiments,letters that appear to be farther than about a second distance thresholdfrom the user are displayed on the system at a second depth that appearsto be farther away than the first depth. In some embodiments, the seconddistance threshold is about 300 cm. In some embodiments, the seconddistance threshold is about 600 cm. In some embodiments, the seconddistance threshold is about 10 m. For example, text viewed on abillboard while driving may be rendered at the second depth.

The difference between the first and second distance thresholds can bedifferent in various embodiments. The magnitude of the difference may bebased on a number of factors, such as, for example, how many depths atwhich the system can display text, the precision or accuracy of thesystem's ability to perceive distances from real-world objects or text,or what the manual or factory setting is. In some embodiments, thedifference is less than 100 m. In some embodiments, the difference isless than 700 cm. In some embodiments, the difference is less than 30cm. In certain embodiments, the difference is zero (e.g., the firstdistance threshold and the second distance threshold are the same).

In some embodiments, it is possible for the system to handle negativedifferences. That is, where there is some overlap in which an object ortext fits the criteria both for being displayed at the first depth andat the second depth. In such embodiments, the wearable system can usecontext clues to determine which depth will provide the most seamlessviewing experience for the user. For example, an object that initiallyappears close to the user but is rapidly moving away from the user mayinitially fit the criteria to be displayed at the first depth. However,the system may determine that because of the trajectory of the object'slocation, it will display the object at the second depth.

Some embodiments of the wearable system are able to display text atthree or more depths. In such cases, intermediate distance thresholds orranges of distances corresponding to third, fourth, etc. depths betweenthe first and second depths can be included. For example, in someembodiments, text may be rendered at a third depth when the lettersappear to be, for example, between about 100 cm and 300 cm away from thedisplay 220.

The wearable system 200 can be configured to identify or recognize textfrom an image automatically or in response to a user input. Inembodiments where text is automatically identified, a user can view animage with text and the system can identify and display the text asdescribed herein without command by the user. In embodiments where thetext is identified in response to a user input, a user can use a varietyof commands to initiate the identification or display of the text. Forexample, a command may be a verbal cue, a hand gesture, a head motion(e.g., nodding), an eye movement (e.g., blinking), etc.

Example Processes of Modifying Display Characteristics of a Text

FIG. 17 illustrates an example process of a sensory eyewear forfacilitating a user's interactions with the environment. The process1700 can be performed by one or more components of the wearable system200 (e.g., by the local processing & data module 260, the remoteprocessing module 270, alone or in combination).

At block 1704, the wearable system can receive an optical signal throughone or more cameras. The one or more cameras may be part of theoutward-facing imaging system 464 or be part of another computing devicesuch as a dash cam or the user input device 466.

At block 1708, the wearable system can include identifying an image fromthe signal. For example, the wearable system can convert the opticalsignal to human readable image. In some embodiments, identifying animage from the signal can also include recognizing the content of theimage such as, e.g., performing an optical character recognition (OCR)on the image using one or more object recognizers 708. In certainembodiments, the optical character recognition process includesidentifying likely candidates for the text or language of the one ormore letters or characters. The optical character recognition processmay use various contextual information (e.g., context clues) to performthe recognition. Some example contextual information may include theactivity engaged in by the user or someone in a vicinity of the user,the geographical location of the user, the current speed of travel ofthe user, the current altitude of the user, the volume or type ofambient noise detected by the system, the level or type of visible orother light in the area detected by the display system, the temperatureor climate detected by the display system, the perceived distance of thecharacters or letters from the user, or the category or genre of wordsdetected by the display.

With continued reference to FIG. 17, the process 1700 can furtherinclude determining whether the image comprises letters or characters asshown in block 1712. In some embodiments, if the process 1700 determinesthat the image does not comprise letters or characters, the process canrevert back to the block 1704. If the process 1700 determines that theimage comprises letters or characters, the method continues onto theblock 1716.

At block 1716, the wearable system can convert the letters or charactersinto text. This can include, for example, displaying the text in asecond language that is different from the first language (as furtherdescribed with reference to FIGS. 18 and 19). In some embodimentsconverting the one or more letters or characters (from an image) into atext can be done in response to receiving an input or command from auser or another person. Such an input or command can include a varietyof modes, such as, for example, a verbal command, a hand gesture, amotion of the head, or a movement of one or more of the user's eyes.These examples should not be viewed as limiting.

At block 1720, the wearable system can instruct a display to render textto appear at a first depth of a plurality of depths from the user. Insome embodiments, displaying the text includes transmitting light to theuser as an image through an optically transmissive eyepiece. Theeyepiece can be any of those described herein. For example, light may bedirected into and eye of the user to form an image in the eye. Thewearable system may use a fiber scanning projector or other projector asdescribed herein. In some embodiments, the method may receive locationdata from a GPS system 37 (described with reference to FIG. 2B). Thislocation data can be used to help the system infer text extracted froman image, as further described herein with reference to FIGS. 16A-16E.

The wearable system can also modify the text and render the modifiedtext (e.g., project light from the display 220 toward the user's eyes).For example, the method can display the text, relative to the originalletters or characters, in a different font, font size, color, backgroundor background color, format, level of clarity, language, or brightness.In some embodiments, the method can include animating the text orincorporating virtual objects that interact with the text.

Examples of Modifying Content of Signage

In addition to or in alternative to modifying the displaycharacteristics of the text, the wearable system can also modify thecontent of the text, such as, e.g., by translating the text from onelanguage to another, and display the modified text. FIG. 18 illustratesan example of assisting a user in understanding signage in a physicalenvironment by modifying the content of the signage where the signage istranslated from a local language to a target language which the user ofthe wearable system is able to understand.

FIG. 18 illustrates two scenes 1800 a and 1800 b. The scene 1800 a canbe perceived by a user without wearing the HMD described herein. Thescene 1800 b can be perceived by the user while wearing the HMD (e.g.,through the display 220, without the translation process to bedescribed). As illustrated, both scenes 1800 a and 1800 b include astreet 1802 and pedestrians 1804. The scene 1800 a also shows streetsigns 1810 a and 1820 a which include Simplified Chinese characters. Thesign 1820 a also includes English characters. However, the user (notshown in FIG. 18) of the HMD may be an English speaker and may notunderstand the Chinese characters. Advantageously, in some embodiments,the wearable system can automatically recognize the text on the streetsigns 1810 a and 1820 b and convert the foreign language text portion ofthe street signs into a language that the user understands. The wearablesystem can also present the translated signage as a virtual image overthe physical signs as shown in the scene 1800 b. Accordingly, the userwould not perceive the Chinese text in the signs 1810 a, 1820 a butinstead would perceive the English text shown in the signs 1810 b, 1820b, because the HMD displays virtual image (with the English text) withsufficient brightness that the underlying Chinese text is not perceived.

The HMD (e.g., the wearable system 200) can use similar techniques asdescribed with reference to FIGS. 16A-17 to identify a sign in theuser's environment and recognize the sign. In some situations, thewearable system 200 may be configured to translate only a portion of asign. For example, the wearable system 200 translates only the portionof the sign 1820 a having Chinese text but not the portion of the sign1820 a having English text (“GOLDSTAR”), because the English portion canbe understood by the user (e.g., because it is in the target language ofthe user). However, in situations where the user is bilingual such thatthe user can read both English and Simplified Chinese, the wearablesystem 200 may be configured not to translate any of the text on thesigns 1810 a and 1820 a into the signs 1810 b and 1820 b.

As described with reference to FIGS. 16A-16E, the wearable system 200can be configured to adjust the display characteristics of the signs.For example, the text resulted from the translation of the Chineseportion of the sign 1820 a may be longer than the original Chinesecharacters on the sign 1820 a. As a result, the wearable system mayreduce the font size of the translated text (e.g., “shopping center”)such that the rendered text (as shown in the sign 1820 b) can fit withinthe boundary of the original sign.

Although FIG. 18 shows Simplified Chinese and English characters, thisis for illustration and is not a limitation. The language recognized andconverted by embodiments of the wearable display system 200 can includeany language such as, e.g., English, Chinese (simplified ortraditional), Japanese, Korean, French, Spanish, German, Russian,Arabic, a Romance language, an Indo-European language, a Sino-Tibetanlanguage, an Afro-Asiatic language, Hebrew, a Malayo-Polynesianlanguage, etc.

Example Processes of Modifying Content of Signage

FIG. 19 illustrates an example process of assisting a user inunderstanding signage in a physical environment. The example process1900 may be performed by one or more components of the wearable system200 (e.g., by the local processing & data module 260, the remoteprocessing module 270, alone or in combination).

At block 1910, the wearable system can receive images of a user'senvironment. The images can be captured by the outward-facing imagingsystem 464, a user input device 466, or a camera on another device thatis external to the wearable system. The images may be still images,frames of a video, or a video.

At block 1920, the wearable system can analyze the images to identifysignage in the user's environment. The wearable system can use theobject recognizer 708 to perform such identification. For example, theobject recognizer 708 can detect the existence of text on the object andthus classify the object as a sign or can recognize the regular boundaryof the signage (e.g., the rectangular signs 1810 a, 1810 b in FIG. 18).

At block 1930, the wearable system can recognize text on the signage1930. For example, the wearable system can determine which characters orletters there are on the sign. As another example, the wearable systemcan determine which language the text is in. The wearable system canmake such determination based on context clues associated with the useror the sign, such as, e.g., the location of the user, the syntax,grammar, spelling of the text, etc. The wearable system can furtherdetermining the meaning of the text (e.g., by looking up in adictionary) at block 1930.

At block 1940, the wearable system can convert at least a portion of thetext to a target language. The target language may be determined basedon the user's preference or the user's demographic information. Forexample, the target language may be the official language associatedwith the user's country of origin, the user's mother tongue, thelanguage most frequently used by the user, or the language that the userhas spoken (e.g., in a voice command to a wearable system or in aconversation with another user), etc. The target language can also beset in accordance with the user's preference. For example, the user mayprefer the signs to be translated into English even though the user'snative tongue is French.

At the optional block 1950, the wearable system can modify the displaycharacteristics associated with the text. For example, the wearablesystem can add focus indicators to the text (or the backgroundassociated with the text), as well as change the font size or color ofthe text. Example modifications of the display characteristics arefurther described with reference to FIGS. 16A-17.

At the optional block 1960, the wearable system can cause the text to berendered in the target language by a mixed reality device. The MR devicemay be the HMD described herein. Where the display characteristics aremodified, the wearable system can also cause the modified displaycharacteristics to be rendered. In situations where only a portion ofthe text is translated into the target language, the wearable system caneither display only the portion of the translated text or display boththe translated portion and the portion of the original text that was nottranslated. The modified text may be rendered over the original text onthe physical signage such that the original text may be occluded fromthe user's view.

Although the examples in FIGS. 18 and 19 are described with reference totranslating text on signage, similar techniques can also be applied totext that is embodied in other types of mediums (such as books,television, computer monitor, etc.).

Additional Aspects Related to Sign Language

Additional aspects of applications of sensory eyewear in a sign languageare further provided below.

In a 1st aspect, a method for providing text converted from signlanguage through an augmented reality system, the method comprising:under the control of an augmented reality (AR) system comprising animaging system: capturing, via the imaging system, image information;detecting gestures in the image information, the gestures beingcandidates for sign language recognition; recognizing sign language inthe detected gestures; converting the recognized sign language to text;and displaying the converted text.

In a 2nd aspect, the method of aspect 1, further comprising: receiving arequest for auxiliary information on the converted text; retrievingauxiliary information associated with the requested converted text;displaying the auxiliary information using the AR system; detecting acondition for dismissing display of the converted text or the auxiliaryinformation; and dismissing display of the converted text or theauxiliary information.

In a 3rd aspect, the method of aspect 2, wherein the condition fordismissing display of the converted text or the auxiliary information isbased on a user interface interaction.

In a 4th aspect, the method of aspect 3, wherein the user interfaceinteraction is based, at least in part, on eye movements of a user ofthe AR system.

In a 5th aspect, the method of any one of aspects 2-4, wherein thecondition for dismissing display of the converted text or the auxiliaryinformation is based, at least in part, on a duration of time.

In a 6th aspect, the method of any one of aspects 2-5, wherein thecondition for dismissing display of the converted text or the auxiliaryinformation is based, at least in part, on conversion of additional signlanguage gestures or reception of additional auxiliary information.

In a 7th aspect, the method of any one of aspects 2-6, furthercomprising: detecting a condition for re-displaying a dismissed displayof the converted text or the auxiliary information; and re-displaying adismissed display of the converted text or the auxiliary information.

In an 8th aspect, the method of any one of aspects 1-7, whereinconverting the recognized sign language to text comprises applying adeep learning technique.

In a 9th aspect, the method of aspect 8, wherein the deep learningtechnique comprises a neural network.

In a 10th aspect, the method of any one of aspects 1-9, wherein the ARsystem uses a sign language dictionary in sign language recognition andtext conversion.

In an 11th aspect, the method of any one of aspects 1-10, wherein the ARsystem recognizes a sign language which is foreign to a user of the ARsystem.

In a 12th aspect, the method of aspect 11, wherein the AR systemrecognizes the sign language by working through a list of candidate signlanguages, the list being prioritized based at least in part on locationof the AR system.

In a 13th aspect, the method of any one of aspects 11-12, wherein the ARsystem recognizes the sign language by working through a list ofcandidate sign languages, the list being prioritized based at least inpart on a spoken language detected in the environment of the AR system.

In a 14th aspect, an augmented reality (AR) apparatus for translating asign language, comprising: an AR display; an imaging system; a datastore configured to store computer-executable instructions and data; anda processor in communication with the data store, wherein thecomputer-executable instructions, when executed, cause the processor to:receive image information captured by the imaging system; detectgestures in the received image or video information; recognize signlanguage in the detected gestures; translate recognized sign language toa language understood by a user of the AR apparatus; and display, usingthe AR display, information associated with the translated signlanguage.

In a 15th aspect, the apparatus of aspect 14, wherein thecomputer-executable instructions, when executed, further cause theprocessor to: receive a request for auxiliary information on thetranslated sign language; retrieve auxiliary information related to therequested sign language; and display, using the AR display, theretrieved auxiliary information.

In a 16th aspect, the apparatus of any of aspects 14-15, wherein theprocessor detects the gestures and recognizes the sign language bytransmitting received image information through a communication networkto a remote processor for the remote processor to detect the gesturesand to recognize the sign language.

In a 17th aspect, the apparatus of any one of aspects 14-16, wherein theimaging system comprises a plurality of cameras or a wide-angle camera.

In an 18th aspect, the apparatus of any one of aspects 14-17, whereinthe processor is further configured to: determine a source of thedetected gestures; and upon determining the source of the detectedgestures to be the user of the AR apparatus, transmit the translatedsign language to another device for display.

In a 19th aspect, the apparatus of any one of aspects 14-18, furthercomprising an audio amplifier, and the processor is further programmedto present the translated sign language in audio through the audioamplifier.

In a 20th aspect, the apparatus of aspect 19, wherein the processor isfurther configured to present the auxiliary information in audio throughthe audio amplifier.

In a 21st aspect, the apparatus of any one of aspects 14-20, wherein thelanguage understood by the user of the AR apparatus comprises a signlanguage different from the recognized sign language.

In a 22nd aspect, an augmented reality (AR) system for facilitatingremote communication involving one or more sign languages, comprising: aplurality of wearable AR devices, each comprising: an AR display; animaging system; and a communication system for communicating over acommunication network; one or more data stores configured to storecomputer-executable instructions and data; and one or more processors incommunication with the data stores, wherein the computer-executableinstructions, when executed, configure the one or more processors to:receive image information captured by the imaging system of a firstwearable AR device in the plurality of wearable AR devices; detect signlanguage in the received image information; convert the detected signlanguage into text; transmit, through the communication network, theconverted text to a second wearable AR device in the plurality ofwearable AR devices; and display, on the AR display of the secondwearable AR device, the converted text.

In a 23rd aspect, the system of aspect 22, wherein the second wearableAR device further displays a world map of the first user.

In a 24th aspect, the system of aspect 23, wherein the world map of thefirst user comprises an avatar of the first user.

In a 25th aspect, the system of any one of aspects 22-24, wherein eachof the plurality of wearable AR devices includes one or more data storesand one or more processors, and the processor functionalities areperformed by a local processor.

In a 26th aspect, a wearable system for sign language recognition, thewearable system comprising: a head-mounted display configured to presentvirtual content to a user; an imaging system configured to image anenvironment of the user; and a hardware processor in communication withthe head-mounted display and the imaging system, and programmed to:receive an image captured by the imaging system; detect a gesture in theimage with an object recognizer; recognize a meaning of the gesture in asign language; identify a target language based on contextualinformation associated with the user; translate the gesture into thetarget language based on the recognized meaning; generate virtualcontent based at least partly on a translation of the gesture into thetarget language; and cause the head-mounted display to render thevirtual content to the user.

In a 27th aspect, the wearable system of aspect 26, wherein the imagingsystem comprises one or more of wide-angle cameras configured to image asurrounding of the user.

In a 28th aspect, the wearable system of any one of aspects 26-27, wherein the hardware processor is further programmed to access auxiliaryinformation associated with the gesture; and where the virtual contentrendered by the head-mounted display comprises the auxiliaryinformation.

In a 29th aspect, the wearable system of any one of aspects 26-28,wherein to identify a target language based on contextual informationassociated with the user, the hardware processor is programmed to: setthe target language as a language understood by the user based on atleast one of: the user's speech as captured by the wearable system, theuser's location, or an input from the user selecting the language as thetarget language.

In a 30th aspect, the wearable system of any one of aspects 26-29,wherein the hardware processor is programmed to determine whether thetarget language is a spoken language; and in response to a determinationthat the target language is a spoken language, to play an audio streamof a speech associated with the translated gesture in the targetlanguage.

In a 31st aspect, the wearable system of any one of aspects 26-29,wherein the hardware processor is programmed to determine whether thetarget language is another sign language; and in response to adetermination that the target language is another sign language, presenta graphic of another gesture in the other sign language as thetranslation of the gesture.

In a 32nd aspect, the wearable system of any one of aspects 26-31,wherein to recognize the meaning of the gesture in the sign language,the hardware processor is programmed to apply a deep neutral networktechnique on at a portion of the image captured by the imaging system.

In a 33rd aspect, the wearable system of any one of aspects 26-32,wherein the hardware processor is further programmed to identify thesign language from a list of candidate sign language based at leastpartly on a location of the user.

In a 34th aspect, the wearable system of any one of aspects 26-33,wherein to translate the gesture into the target language based on therecognized meaning, the hardware processor is programmed to convert thegesture to a text expression in the target language.

In a 35th aspect, the wearable system of any one of aspects 26-34,wherein the hardware processor is programmed to determine a source ofthe detected gesture; and upon determining the source of the detectedgesture to be the user of the wearable system, communicate thetranslation of the gesture in the target language to a wearable deviceof another user.

In a 36th aspect, the wearable system of any one of aspects 26-35,wherein the hardware processor is programmed to detect a condition fromdismissing the virtual content from display by the head-mounted display,and remove the virtual content from the display by the head-mounteddisplay in response to a detection of the condition.

In a 37th aspect, the wearable system of aspect 36, wherein thecondition comprises at least one of: a duration of time, a user's handgesture, or an input from a user input device.

In a 38th aspect, the wearable system of any one of aspects 26-37,wherein the image comprises one or more frames of a video.

In a 39th aspect, a method for sign language recognition, the methodcomprising: receiving an image captured by an imaging system; analyzingthe image to detect a gesture of a user; detecting a presence of acommunication in a sign language based at least partly on the detectedgesture; recognizing a meaning of the gesture in the sign language;identifying a target language to which the gesture will be translatedinto; translating the gesture into the target language based on therecognized meaning; generating virtual content based at least partly ona translation of the gesture into the target language; and causing ahead-mounted display to render the virtual content to the user.

In a 40th aspect, the method of aspect 39, wherein the image is receivedfrom a first wearable device configured to present mixed reality contentwhile the virtual content is communicated to a second wearable devicefor rendering, wherein the first wearable device and the second wearabledevice are configured to present mixed reality content to the user.

In a 41st aspect, the method of aspect 39, wherein translating thegesture into the target language based on the recognized meaningcomprises converting the gesture to a text expression in the targetlanguage.

In a 42nd aspect, the method of any one of aspects 39-41, wherein thevirtual content comprises a text expression in the target language or agraphic illustrating another in the target language.

In a 43rd aspect, the method of any one of aspects 39-42, whereinrecognizing the meaning of the gesture in the sign language comprisesapplying a deep neutral network technique on at a portion of the imagecaptured by the imaging system.

In a 44th aspect, the method of any one of aspects 39-43, whereindetecting the presence of the communication in the sign languagecomprises: identifying the sign language from a list of candidate signlanguages; and determining the detected gesture correspond to anexpression in the sign language.

In a 45th aspect, the method of aspect 44, wherein determining thedetected gesture correspond to an expression in the sign languagecomprises analyzing the gesture in connection with the lip movements ofa person making the gesture and audio data captures while the user ismaking the gesture.

Additional Aspects Related to Text Modification

Additional aspects modifying characteristics of a text by a sensoryeyewear are further described below.

In a 1st aspect, a head-mounted display device configured to projectaugmented reality image content, the display device comprising: a frameconfigured to be wearable on a head of the user and configured tosupport a display in front of an eye of the user; one or more camerasconfigured to receive an optical signal; processing electronicsconfigured to: receive a signal from the one or more cameras; identifyan image from the signal; determine whether the image comprises text(e.g., one or more letters or characters); convert the text intomodified text; and instruct the display to render the modified text.

In a 2nd aspect, the head-mounted display device of aspect 1, whereinthe display comprises one or more light sources and one or morewaveguide stacks configured to direct light into an eye of the user toform images in the eye.

In a 3rd aspect, the head-mounted display device of aspect 2, whereinthe one or more light sources is configured to direct light into thewaveguide stacks.

In a 4th aspect, the head-mounted display device of any one of aspects2-3, wherein the one or more light sources comprises a fiber scanningprojector.

In a 5th aspect, the head-mounted display device of any one of aspects1-4, wherein the one or more cameras comprises one or more videocameras.

In a 6th aspect, the head-mounted display device of any one of aspects1-5, wherein the processing electronics are configured to use an opticalcharacter recognition algorithm to convert one or more letters orcharacters in the image into text.

In a 7th aspect, the head-mounted display device of aspect 6, whereinthe processing electronics are configured to access a database toidentify likely candidates for the text or language of the one or moreletters or characters.

In an 8th aspect, the head-mounted display device of any one of aspects6-7, wherein the processing electronics is configured to receive inputassociated with one or more of an activity engaged in by the user, ageographical location of the user, a speed of travel of the user, analtitude of the user, a volume or type of ambient noise detected by thedisplay, a level or type of visible or other light in the area detectedby the display, a temperature or climate detected by the display, aperceived distance of the text from the user, or a category of wordsdetected by the display.

In a 9th aspect, the head-mounted display device of any one of aspects1-8, further comprising a GPS system.

In a 10th aspect, the head-mounted display device of any one of aspects1-9, wherein the modified text is in a second font size different from afirst font size of the text. The second font size can be larger than thefirst font size.

In an 11th aspect, the head-mounted display device of any one of aspects1-10, wherein the modified text is more legible to the user then thetext.

In a 12th aspect, the head-mounted display device of any one of aspects1-11, wherein the processing electronics are configured to add graphicalelements to the text in part to form the modified text.

In a 13th aspect, the head-mounted display device of any one of aspects1-12, wherein the processing electronics are configured to display theone or more letters or characters of the text in a second font differentfrom a first font of the one or more letters or characters.

In a 14th aspect, the head-mounted display device of any one of aspects1-13, wherein the processing electronics are configured to magnify theone or more letters or characters of the text relative to what the userwould see without the head-mounted display.

In a 15th aspect, the head-mounted display device of any one of aspects1-14, wherein the processing electronics are configured to display aborder region, the border region bounding an inner region.

In a 16th aspect, the head-mounted display device of aspect 15, whereinthe processing electronics are configured to display the one or moreletters or characters within the inner region.

In a 17the aspect, the head-mounted display device of any one of aspects1-16, wherein the processing electronics are configured to display theone or more letters or characters of the text against a secondbackground different from a first background against which the userwould read the one or more letters or characters without thehead-mounted display.

In an 18th aspect, the head-mounted display device of aspect 17, whereinthe second background comprises a monochrome background.

In a 19th aspect, the head-mounted display device of aspect 18, whereinthe monochrome background comprises white.

In a 20th aspect, the head-mounted display device of any one of aspects17-19, wherein the first background comprises what the user would seewithout the head-mounted display.

In a 21st aspect, the head-mounted display device of any one of aspects1-20, wherein the text is adapted to be editable by a text editor.

Although aspects 1-21 are described with reference to a head-mounteddisplay, similar functions described in these aspects can also beimplemented with a head-mounted device or the wearable system describedwith reference to FIG. 2A. Further, the display can comprise a pluralityof depth planes and the head-mounted device is configured to identify adepth plane to render the modified text based at least partly on theuser's eye capacity.

In a 22nd aspect, a method for projecting augmented reality imagecontent using a head-mounted display, the method comprising: undercontrol of a hardware processor: receiving an optical signal from one ormore cameras; using an optical character recognition module, identifyingan image from the signal; determining whether the image comprises one ormore letters or characters; converting the one or more letters orcharacters into text; and displaying the text on the head-mounteddisplay, wherein displaying the text comprises transmitting light to theuser as an image through an optically transmissive eyepiece.

In a 23rd aspect, the method of aspect 22, further comprising directinglight into an eye of the user to form images in the eye.

In a 24th aspect, the method of any one of aspects 22-23, furthercomprising directing light into the eyepiece using a fiber scanningprojector.

In a 25th aspect, the method of any one of aspects 22-24, wherein usingan optical character recognition module comprises identifying likelycandidates for the text or language of the one or more letters orcharacters.

In a 26th aspect, the method of any one of aspects 22-25, wherein usingan optical character recognition module comprises receiving an inputcomprising information associated with one or more of an activityengaged in by the user, a geographical location of the user, a speed oftravel of the user, an altitude of the user, a volume or type of ambientnoise detected by the display, a level or type of visible or other lightin the area detected by the display, a temperature or climate detectedby the display, a perceived distance of the one or more letters orcharacters from the user, or a category of words detected by thedisplay.

In a 27th aspect, the method of any of aspects 22-26, wherein convertingthe one or more letters or characters into text comprises displaying thetext in a second language different from a first language associatedwith the one or more letters or characters. In some implementations ofthe 27th aspect, the method includes translating the text into thesecond language.

In a 28th aspect, the method of any of one of aspects 22-27, furthercomprising receiving location data from a GPS system.

In a 29th aspect, the method of any one of aspects 22-28, whereindisplaying the one or more letters or characters on the head-mounteddisplay comprises displaying the one or more letters or characters in asecond font size different from a first font size of the one or moreletters or characters.

In a 30th aspect, the method of any one of aspects 22-29, whereindisplaying the one or more letters or characters on the head-mounteddisplay comprises displaying the one or more letters or characters morelegibly to the user than without the head-mounted display.

In a 31st aspect, the method of any one of aspects 22-30, whereindisplaying the one or more letters or characters on the head-mounteddisplay comprises displaying the one or more letters or characters in afont size that is larger than would appear to the user without thehead-mounted display.

In a 32nd aspect, the method of any one of aspects 22-31, whereindisplaying the one or more letters or characters on the head-mounteddisplay comprises displaying the one or more letters or characters in asecond font different from a first font of the one or more letters orcharacters.

In a 33th aspect, the method of any one of aspects 22-32, whereindisplaying the one or more letters or characters on the head-mounteddisplay comprises magnifying the one or more letters or charactersrelative to what the user would see without the head-mounted display.

In a 34th aspect, the method of any one of aspects 22-33, whereindisplaying the one or more letters or characters on the head-mounteddisplay comprises displaying a border region, the border region boundingan inner region.

In a 35th aspect, the method of aspect 34, wherein displaying the one ormore letters or characters on the head-mounted display comprisesdisplaying the one or more letters or characters within the innerregion.

In a 36th aspect, the method of any one of aspects 22-35, whereindisplaying the one or more letters or characters on the head-mounteddisplay comprises displaying the one or more letters or charactersagainst a second background different from a first background againstwhich the user would read the one or more letters or characters withoutthe head-mounted display.

In a 37th aspect, the method of aspect 36, wherein the second backgroundcomprises a monochrome background.

In a 38th aspect, the method of aspect 37, wherein the monochromebackground comprises white.

In a 39th aspect, the method of any of one of aspects 36-38, wherein thefirst background comprises what the user would see without thehead-mounted display.

In a 40th aspect, the method of any one of aspects 22-39, wherein thetext is adapted to be editable by a text editor.

In a 41st aspect, the method of any one of aspects 22-40, whereinconverting the one or more letters or characters into text comprisesreceiving an input from a user.

In a 42nd aspect, the method of aspect 41, wherein receiving an inputfrom a user comprises receiving one or more of a verbal command, a handgesture, a motion of the head, or a movement of one or more of theuser's eyes.

In a 43rd aspect, the method of any one of aspects 22-42, wherein thetext is displayed at a first depth that appears to be closer than asecond depth if the one or more letters or characters appear to becloser than a first distance threshold.

In a 44th aspect, the method of any one of aspects 22-43, wherein thetext is displayed at a second depth that appears to be farther away thana first depth if the one or more letters or characters appear to befarther away than a second distance threshold.

In a 45th aspect, the method of any one of aspects 43-44, wherein thetext is displayed at a third depth that appears to be farther away thanthe first depth but closer than the second depth if the one or moreletters or characters appear to be farther away than a first distancethreshold and closer than the second distance threshold.

In a 46th aspect, the method of any one of aspects 43-45, wherein thefirst distance threshold is 80 cm.

In a 47th aspect, the method of any one of aspects 43-46, wherein thesecond distance threshold is 600 cm.

In a 48th aspect, the method of any one of aspects 43-47, wherein thedifference between the second distance threshold and first distancethreshold is less than 100 m.

Additional Aspects Related to Signage Modification

In a 1st aspect, an augmented reality system comprising: anoutward-facing imaging system; non-transitory memory configured to storeimages obtained by the outward-facing imaging system; and a hardwareprocessor programmed to: receive images of an environment of a user ofthe augmented reality system obtained by the outward-facing imagingsystem; analyze the images to identify signage in the user'senvironment; recognize text on the signage; convert at least a portionof the text to a target language; and instruct a display to render theconverted text to the user.

In a 2nd aspect, the augmented reality system of aspect 1, wherein thehardware processor is programmed to modify display characteristicsassociated with the text.

In a 3rd aspect, the augmented reality system of aspect 1 or 2, whereinto convert at least a portion of the text to a target language, thehardware processor is programmed to identify a language of the text onthe signage and to convert the language to the target language.

In a 4th aspect, the augmented reality system of any one of aspects 1-3,wherein the hardware processor is programmed to determine the targetlanguage based at least in part on a location of the user.

In a 5th aspect, the augmented reality system of any one of aspects 1-4,wherein to recognize text on the signage, the hardware processor isprogrammed to recognize text that is in the target language.

In a 6th aspect, the augmented reality system of aspect 5, wherein thehardware processor is programmed not to convert the text that is in thetarget language.

Other Considerations

Each of the processes, methods, and algorithms described herein ordepicted in the attached figures may be embodied in, and fully orpartially automated by, code modules executed by one or more physicalcomputing systems, hardware computer processors, application-specificcircuitry, or electronic hardware configured to execute specific andparticular computer instructions. For example, computing systems caninclude general purpose computers (e.g., servers) programmed withspecific computer instructions or special purpose computers, specialpurpose circuitry, and so forth. A code module may be compiled andlinked into an executable program, installed in a dynamic link library,or may be written in an interpreted programming language. In someimplementations, particular operations and methods may be performed bycircuitry that is specific to a given function.

Further, certain implementations of the functionality of the presentdisclosure are sufficiently mathematically, computationally, ortechnically complex that application-specific hardware or one or morephysical computing devices (utilizing appropriate specialized executableinstructions) may be necessary to perform the functionality, forexample, due to the volume or complexity of the calculations involved orto provide results substantially in real-time. For example, animationsor video may include many frames, with each frame having millions ofpixels, and specifically programmed computer hardware is necessary toprocess the video data to provide a desired image processing task orapplication in a commercially reasonable amount of time.

Code modules or any type of data may be stored on any type ofnon-transitory computer-readable medium, such as physical computerstorage including hard drives, solid state memory, random access memory(RAM), read only memory (ROM), optical disc, volatile or non-volatilestorage, combinations of the same or the like. The methods and modules(or data) may also be transmitted as generated data signals (e.g., aspart of a carrier wave or other analog or digital propagated signal) ona variety of computer-readable transmission mediums, includingwireless-based and wired/cable-based mediums, and may take a variety offorms (e.g., as part of a single or multiplexed analog signal, or asmultiple discrete digital packets or frames). The results of thedisclosed processes or process steps may be stored, persistently orotherwise, in any type of non-transitory, tangible computer storage ormay be communicated via a computer-readable transmission medium.

Any processes, blocks, states, steps, or functionalities in flowdiagrams described herein or depicted in the attached figures should beunderstood as potentially representing code modules, segments, orportions of code which include one or more executable instructions forimplementing specific functions (e.g., logical or arithmetical) or stepsin the process. The various processes, blocks, states, steps, orfunctionalities can be combined, rearranged, added to, deleted from,modified, or otherwise changed from the illustrative examples providedherein. In some embodiments, additional or different computing systemsor code modules may perform some or all of the functionalities describedherein. The methods and processes described herein are also not limitedto any particular sequence, and the blocks, steps, or states relatingthereto can be performed in other sequences that are appropriate, forexample, in serial, in parallel, or in some other manner. Tasks orevents may be added to or removed from the disclosed exampleembodiments. Moreover, the separation of various system components inthe implementations described herein is for illustrative purposes andshould not be understood as requiring such separation in allimplementations. It should be understood that the described programcomponents, methods, and systems can generally be integrated together ina single computer product or packaged into multiple computer products.Many implementation variations are possible.

The processes, methods, and systems may be implemented in a network (ordistributed) computing environment. Network environments includeenterprise-wide computer networks, intranets, local area networks (LAN),wide area networks (WAN), personal area networks (PAN), cloud computingnetworks, crowd-sourced computing networks, the Internet, and the WorldWide Web. The network may be a wired or a wireless network or any othertype of communication network.

The systems and methods of the disclosure each have several innovativeaspects, no single one of which is solely responsible or required forthe desirable attributes disclosed herein. The various features andprocesses described above may be used independently of one another, ormay be combined in various ways. All possible combinations andsubcombinations are intended to fall within the scope of thisdisclosure. Various modifications to the implementations described inthis disclosure may be readily apparent to those skilled in the art, andthe generic principles defined herein may be applied to otherimplementations without departing from the spirit or scope of thisdisclosure. Thus, the claims are not intended to be limited to theimplementations or embodiments shown herein, but are to be accorded thewidest scope consistent with this disclosure, the principles and thenovel features disclosed herein.

Certain features that are described in this specification in the contextof separate implementations or embodiments also can be implemented incombination in a single implementation or embodiment. Conversely,various features that are described in the context of a singleimplementation or embodiment also can be implemented in multipleimplementations or embodiments separately or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a subcombination. No single feature orgroup of features is necessary or indispensable to each and everyembodiment.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “e.g.,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements or steps.Thus, such conditional language is not generally intended to imply thatfeatures, elements or steps are in any way required for one or moreembodiments or that one or more embodiments necessarily include logicfor deciding, with or without author input or prompting, whether thesefeatures, elements or steps are included or are to be performed in anyparticular embodiment. The terms “comprising,” “including,” “having,”and the like are synonymous and are used inclusively, in an open-endedfashion, and do not exclude additional elements, features, acts,operations, and so forth. Also, the term “or” is used in its inclusivesense (and not in its exclusive sense) so that when used, for example,to connect a list of elements, the term “or” means one, some, or all ofthe elements in the list. In addition, the articles “a,” “an,” and “the”as used in this application and the appended claims are to be construedto mean “one or more” or “at least one” unless specified otherwise.

As used herein, a phrase referring to “at least one of” a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: A, B, or C” is intended to cover: A, B, C,A and B, A and C, B and C, and A, B, and C. Conjunctive language such asthe phrase “at least one of X, Y and Z,” unless specifically statedotherwise, is otherwise understood with the context as used in generalto convey that an item, term, etc. may be at least one of X, Y or Z.Thus, such conjunctive language is not generally intended to imply thatcertain embodiments require at least one of X, at least one of Y and atleast one of Z to each be present.

Similarly, while operations may be depicted in the drawings in aparticular order, it is to be recognized that such operations need notbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. Further, the drawings may schematically depict one more exampleprocesses in the form of a flowchart. However, other operations that arenot depicted can be incorporated in the example methods and processesthat are schematically illustrated. For example, one or more additionaloperations can be performed before, after, simultaneously, or betweenany of the illustrated operations. Additionally, the operations may berearranged or reordered in other implementations. In certaincircumstances, multitasking and parallel processing may be advantageous.Moreover, the separation of various system components in theimplementations described above should not be understood as requiringsuch separation in all implementations, and it should be understood thatthe described program components and systems can generally be integratedtogether in a single software product or packaged into multiple softwareproducts. Additionally, other implementations are within the scope ofthe following claims. In some cases, the actions recited in the claimscan be performed in a different order and still achieve desirableresults.

What is claimed is:
 1. A head-mounted display device configured toproject augmented reality image content, the display device comprising:a frame configured to be wearable on a head of a user and configured tosupport a display in front of an eye of the user; one or more camerasconfigured to generate an optical signal; processing electronicsconfigured to: receive the optical signal from the one or more cameras;identify, from the optical signal, at least one physical object in anenvironment of the user; determine whether the at least one physicalobject comprises text; in response to determining that the at least onephysical object comprises text, determine a modification to be appliedto the text; generate modified text based at least on the determinedmodification and the text; generate a virtual representation of the atleast one physical object comprising the modified text; instruct thedisplay to render the virtual representation of the at least onephysical object in the environment of the user to occlude the at leastone physical object
 2. The head-mounted display device of claim 1,wherein the display comprises one or more light sources and one or morewaveguide stacks configured to direct light into an eye of the user toform images in the eye.
 3. The head-mounted display device of claim 2,wherein the one or more light sources is configured to direct light intothe waveguide stacks.
 4. The head-mounted display device of claim 2,wherein the one or more light sources comprises a fiber-scanningprojector.
 5. The head-mounted display device of claim 1, wherein theone or more cameras comprises one or more video cameras.
 6. Thehead-mounted display device of claim 1 wherein the processingelectronics are configured to use an optical character recognitionalgorithm to convert one or more letters or characters in the opticalsignal into text.
 7. The head-mounted display device of claim 6, whereinthe processing electronics are configured to access a database toidentify likely candidates for the text or language of the one or moreletters or characters.
 8. The head-mounted display device of claim 6,wherein the processing electronics is configured to receive inputassociated with one or more of an activity engaged in by the user, ageographical location of the user, a speed of travel of the user, analtitude of the user, a volume or type of ambient noise detected by thedisplay, a level or type of visible or other light in the area detectedby the display, a temperature or climate detected by the display, aperceived distance of the text from the user, or a category of wordsdetected by the display.
 9. The head-mounted display device of claim 1,further comprising a GPS system.
 10. The head-mounted display device ofclaim 1, wherein the modified text is in a second font size differentfrom a first font size of the text.
 11. The head-mounted display deviceof claim 1, wherein the modified text is more legible to the user thenthe text.
 12. The head-mounted display device of claim 1, wherein themodified text comprises graphical elements added to the text.
 13. Thehead-mounted display device of claim 1, wherein the modified textcomprises the text having one or more letters or characters of the textin a second font different from a first font of the one or more lettersor characters of the text.
 14. The head-mounted display device of claim1, wherein the processing electronics are configured to magnify one ormore letters or characters of the text relative to what the user wouldsee without the head-mounted display.
 15. The head-mounted displaydevice of claim 1, wherein the virtual representation comprises a borderregion bounding an inner region.
 16. The head-mounted display device ofclaim 15, wherein the processing electronics are configured to displayone or more letters or characters of the modified text within the innerregion.
 17. The head-mounted display device of claim 16, wherein theprocessing electronics are configured to display the one or more lettersor characters of the text against a second background different from afirst background against which the user would read the one or moreletters or characters without the head-mounted display.
 18. Thehead-mounted display device of claim 17, wherein the second backgroundcomprises a monochrome background.
 19. The head-mounted display deviceof claim 18, wherein the monochrome background comprises white.
 20. Thehead-mounted display device of claim 17, wherein the first backgroundcomprises what the user would see without the head-mounted display. 21.The head-mounted display device of claim 1, wherein the text is adaptedto be editable by a text editor.
 22. The head-mounted display device ofclaim 1, wherein the virtual representation is displayed at a firstdepth that appears to be closer to the user than a second depth if theone or more letters or characters of the text appear to be closer than afirst distance threshold.
 23. The head-mounted display device of claim1, wherein the virtual representation is displayed at a second depththat appears to be farther away from the user than a first depth if theone or more letters or characters appear to be farther away than asecond distance threshold.
 24. The head-mounted display device of claim22, wherein the text is displayed at a third depth that appears to befarther away from the user than the first depth but closer to the userthan the second depth if the one or more letters or characters appear tobe farther away than a first distance threshold and closer than thesecond distance threshold.